May 20 15:30:02.572447 ip-10-0-134-233 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 20 15:30:02.572458 ip-10-0-134-233 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 20 15:30:02.572465 ip-10-0-134-233 systemd[1]: kubelet.service: Failed with result 'resources'.
May 20 15:30:02.572773 ip-10-0-134-233 systemd[1]: Failed to start Kubernetes Kubelet.
May 20 15:30:13.852107 ip-10-0-134-233 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 20 15:30:13.852126 ip-10-0-134-233 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 646a0745a33a4575858a87f36e221f14 --
May 20 15:32:37.200406 ip-10-0-134-233 systemd[1]: Starting Kubernetes Kubelet...
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.609045    2556 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612632    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612641    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612645    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612648    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612651    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612654    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612672    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:32:37.756053 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612676    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:32:37.715309 ip-10-0-134-233 systemd[1]: Started Kubernetes Kubelet.
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612679    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612682    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612685    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612688    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612690    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612693    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612696    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612698    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612701    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612703    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612708    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612712    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612715    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612719    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612722    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612725    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612727    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612730    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:32:37.757328 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612733    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612735    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612738    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612741    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612743    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612746    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612748    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612751    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612754    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612756    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612759    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612761    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612764    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612766    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612769    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612772    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612774    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612776    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612779    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612781    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:32:37.758285 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612789    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612792    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612794    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612797    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612800    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612803    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612805    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612808    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612811    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612815    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612819    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612821    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612824    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612826    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612829    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612831    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612834    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612837    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612840    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:32:37.758950 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612842    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612846    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612848    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612851    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612854    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612856    2556 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612859    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612861    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612864    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612868    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612870    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612873    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612875    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612877    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612880    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612883    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612886    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612888    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612891    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612893    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:32:37.759949 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.612896    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613268    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613273    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613276    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613279    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613282    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613285    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613288    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613290    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613292    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613295    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613297    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613300    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613303    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613306    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613308    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613311    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613313    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613316    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613318    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:32:37.760896 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613321    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613324    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613326    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613328    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613331    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613334    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613336    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613339    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613342    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613344    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613347    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613350    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613352    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613355    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613357    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613360    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613362    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613365    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613367    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613370    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:32:37.761649 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613372    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613375    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613378    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613380    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613383    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613385    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613388    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613391    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613393    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613396    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613398    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613400    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613403    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613409    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613412    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613416    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613419    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613422    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613425    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:32:37.762330 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613427    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613430    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613432    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613435    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613437    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613440    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613442    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613446    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613449    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613451    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613453    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613463    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613468    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613470    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613473    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613475    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613479    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613483    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613485    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:32:37.763153 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613488    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613492    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613494    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613498    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613500    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613503    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613505    2556 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613509    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.613511    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614161    2556 flags.go:64] FLAG: --address="0.0.0.0"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614170    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614176    2556 flags.go:64] FLAG: --anonymous-auth="true"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614180    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614184    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614187    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614191    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614195    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614198    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614201    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614204    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614209    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614212    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 15:32:37.763784 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614215    2556 flags.go:64] FLAG: --cgroup-root=""
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614217    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614220    2556 flags.go:64] FLAG: --client-ca-file=""
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614223    2556 flags.go:64] FLAG: --cloud-config=""
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614226    2556 flags.go:64] FLAG: --cloud-provider="external"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614229    2556 flags.go:64] FLAG: --cluster-dns="[]"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614233    2556 flags.go:64] FLAG: --cluster-domain=""
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614236    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614239    2556 flags.go:64] FLAG: --config-dir=""
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614241    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614245    2556 flags.go:64] FLAG: --container-log-max-files="5"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614249    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614252    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614255    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614258    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614261    2556 flags.go:64] FLAG: --contention-profiling="false"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614264    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614269    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614272    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614275    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614279    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614282    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614285    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614288    2556 flags.go:64] FLAG: --enable-load-reader="false"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614291    2556 flags.go:64] FLAG: --enable-server="true"
May 20 15:32:37.994520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614293    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614297    2556 flags.go:64] FLAG: --event-burst="100"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614300    2556 flags.go:64] FLAG: --event-qps="50"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614303    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614306    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614310    2556 flags.go:64] FLAG: --eviction-hard=""
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614314    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614316    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614319    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614322    2556 flags.go:64] FLAG: --eviction-soft=""
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614325    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614328    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614331    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614333    2556 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614336    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614339    2556 flags.go:64] FLAG: --fail-swap-on="true"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614342    2556 flags.go:64] FLAG: --feature-gates=""
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614345    2556 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614348    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614351    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614354    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614357    2556 flags.go:64] FLAG: --healthz-port="10248"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614360    2556 flags.go:64] FLAG: --help="false"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614363    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-134-233.ec2.internal"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614367    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 15:32:37.995397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614370    2556 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614373    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614376    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614379    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614382    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614385    2556 flags.go:64] FLAG: --image-service-endpoint=""
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614388    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614391    2556 flags.go:64] FLAG: --kube-api-burst="100"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614394    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614397    2556 flags.go:64] FLAG: --kube-api-qps="50"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614400    2556 flags.go:64] FLAG: --kube-reserved=""
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614403    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614406    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614409    2556 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614412    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614415    2556 flags.go:64] FLAG: --lock-file=""
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614418    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614420    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614423    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614428    2556 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614431    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614434    2556 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614437    2556 flags.go:64] FLAG: --logging-format="text"
May 20 15:32:37.996266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614440    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614443    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614446    2556 flags.go:64] FLAG: --manifest-url=""
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614449    2556 flags.go:64] FLAG: --manifest-url-header=""
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614453    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614456    2556 flags.go:64] FLAG: --max-open-files="1000000"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614460    2556 flags.go:64] FLAG: --max-pods="110"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614463    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614467    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614470    2556 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614473    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614476    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614479    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614482    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614489    2556 flags.go:64] FLAG: --node-status-max-images="50"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614492    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614494    2556 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614497    2556 flags.go:64] FLAG: --pod-cidr=""
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614500    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614506    2556 flags.go:64] FLAG: --pod-manifest-path=""
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614508    2556 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614512    2556 flags.go:64] FLAG: --pods-per-core="0"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614515    2556 flags.go:64] FLAG: --port="10250"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614518    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 15:32:37.997509 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614521    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05dc22dd50dce3ec3"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614524    2556 flags.go:64] FLAG: --qos-reserved=""
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614527    2556 flags.go:64] FLAG: --read-only-port="10255"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614530    2556 flags.go:64] FLAG: --register-node="true"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614533    2556 flags.go:64] FLAG: --register-schedulable="true"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614536    2556 flags.go:64] FLAG: --register-with-taints=""
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614539    2556 flags.go:64] FLAG: --registry-burst="10"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614542    2556 flags.go:64] FLAG: --registry-qps="5"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614545    2556 flags.go:64] FLAG: --reserved-cpus=""
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614548    2556 flags.go:64] FLAG: --reserved-memory=""
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614552    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614555    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614558    2556 flags.go:64] FLAG: --rotate-certificates="false"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614560    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614564    2556 flags.go:64] FLAG: --runonce="false"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614566    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614569    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614574    2556 flags.go:64] FLAG: --seccomp-default="false"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614577    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614580    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614583    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614586    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614589    2556 flags.go:64] FLAG: --storage-driver-password="root"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614592    2556 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614594    2556 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614597    2556 flags.go:64] FLAG: --storage-driver-user="root"
May 20 15:32:37.998646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614600    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614603    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614606    2556 flags.go:64] FLAG: --system-cgroups=""
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614610    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614615    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614618    2556 flags.go:64] FLAG: --tls-cert-file=""
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614621    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614626    2556 flags.go:64] FLAG: --tls-min-version=""
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614628    2556 flags.go:64] FLAG: --tls-private-key-file=""
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614631    2556 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614634    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614637    2556 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614640    2556 flags.go:64] FLAG: --v="2"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614644    2556 flags.go:64] FLAG: --version="false"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614648    2556 flags.go:64] FLAG: --vmodule=""
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614651    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614655    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614738    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614741    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614744    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614746    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614750    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614752    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:32:37.999889 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614756    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614758    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614761    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614764    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614766    2556 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614769    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614771    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614774    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614776    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614779    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614782    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614784    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614788    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614790    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614793    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614795    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614798    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614800    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614803    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614806    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:32:38.001958 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614808    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614811    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614813    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614816    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614819    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614821    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614824    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614826    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614829    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614831    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614834    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614837    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614841    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614844    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614846    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614849    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614852    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614854    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614857    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614859    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:32:38.003038 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614862    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614865    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614868    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614870    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614874    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614876    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614879    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614882    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614884    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614887    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614889    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614892    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614894    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614898    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614902    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614904    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614907    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614909    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614912    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614914    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:32:38.005127 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614917    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614919    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614922    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614924    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614928    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614931    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614933    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614936    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614939    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614941    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614945    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614947    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614949    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614952    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614956    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614960    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614964    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614967    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614970    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:32:38.005993 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.614972    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.614980    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.622365    2556 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.622383    2556 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622444    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622451    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622456    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622461    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622465    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622468    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622473    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622477    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622481    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622485    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622489    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622493    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:32:38.274470 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622497    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622500    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622504    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622508    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622512    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622515    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622521    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622525    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622528    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622532    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622536    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622540    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622544    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622548    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622554    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622558    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622562    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622565    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622569    2556 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:32:38.275308 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622574    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622578    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622582    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622586    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622590    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622595    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622601    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622606    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622609    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622613    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622616    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622620    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622624    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622628    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622632    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622636    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622640    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622644    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622648    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:32:38.279191 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622652    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622655    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622659    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622663    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622667    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622670    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622675    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622679    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622684    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622687    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622692    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622696    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622700    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622704    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622708    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622712    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622715    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622719    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622723    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622727    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:32:38.279737 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622731    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622735    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622739    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622742    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622746    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622749    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622753    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622757    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622761    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622765    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622769    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622773    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622776    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622780    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622783    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:32:38.280290 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622788    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.622796    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622924    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622930    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622934    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622939    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622944    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622947    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622951    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622954    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622959    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622963    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622967    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622971    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622975    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:32:38.280785 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622979    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622982    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622986    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622989    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622993    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.622996    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623000    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623004    2556 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623008    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623014    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623017    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623021    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623025    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623029    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623032    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623036    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623040    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623044    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623048    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623052    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:32:38.538563 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623055    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623059    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623064    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623068    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623072    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623075    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623079    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623083    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623086    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623090    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623094    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623113    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623117    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623121    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623125    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623129    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623133    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623137    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623140    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:32:38.541132 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623145    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623150    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623156    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623161    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623165    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623169    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623173    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623177    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623181    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623184    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623188    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623191    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623195    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623198    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623202    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623207    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623211    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623215    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623218    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623222    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:32:38.542047 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623226    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623229    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623233    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623237    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623242    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623246    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623249    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623252    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623258    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623263    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623267    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623271    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623275    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:37.623280    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.623288    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:32:38.543306 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.623927    2556 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.627779    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.628561    2556 server.go:1019] "Starting client certificate rotation"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.628659    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.628696    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.649966    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.656744    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.672676    2556 log.go:25] "Validated CRI v1 runtime API"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.678033    2556 log.go:25] "Validated CRI v1 image API"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.680883    2556 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.685040    2556 fs.go:135] Filesystem UUIDs: map[34de531f-d838-43f4-9ae1-c69e9d09bbee:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8cb9a30a-2f12-4d32-976b-d7b8a8d72634:/dev/nvme0n1p3]
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.685054    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 15:32:38.544812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.688384    2556 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.689149    2556 manager.go:217] Machine: {Timestamp:2026-05-20 15:32:37.688670413 +0000 UTC m=+0.379926332 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102280 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec280430c3455ed7772c6b5c2934059c SystemUUID:ec280430-c345-5ed7-772c-6b5c2934059c BootID:646a0745-a33a-4575-858a-87f36e221f14 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ce:84:93:79:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ce:84:93:79:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:5f:70:68:5c:7e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.689971    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.690039    2556 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.691825    2556 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.691851    2556 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.691972    2556 topology_manager.go:138] "Creating topology manager with none policy"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.691981    2556 container_manager_linux.go:306] "Creating device plugin manager"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.691988    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.692849    2556 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.695615    2556 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.695730    2556 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.697744    2556 kubelet.go:491] "Attempting to sync node with API server"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.697756    2556 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.697769    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.697777    2556 kubelet.go:397] "Adding apiserver pod source"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.697784    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.698829    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:32:38.545193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.698844    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.702251    2556 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.704953    2556 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705792    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705805    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705811    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705816    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705823    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705829    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705834    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705840    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705846    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705851    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.705859    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.706234    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.706992    2556 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.708028    2556 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.708123    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.708153    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.710362    2556 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-233.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.711844    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-slhbj"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.714701    2556 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.714748    2556 server.go:1295] "Started kubelet"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.714821    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.714859    2556 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.714925    2556 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.718509    2556 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.719604    2556 server.go:317] "Adding debug handlers to kubelet server"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.720737    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-slhbj"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.721961    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-233.ec2.internal.18b14fe17f9435fc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-233.ec2.internal,UID:ip-10-0-134-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-233.ec2.internal,},FirstTimestamp:2026-05-20 15:32:37.71471206 +0000 UTC m=+0.405967980,LastTimestamp:2026-05-20 15:32:37.71471206 +0000 UTC m=+0.405967980,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-233.ec2.internal,}"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.725205    2556 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.727738    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.728447    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.728605    2556 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.728613    2556 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.728687    2556 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.728748    2556 reconstruct.go:97] "Volume reconstruction finished"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.728753    2556 reconciler.go:26] "Reconciler: start to sync state"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.729039    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.729199    2556 factory.go:55] Registering systemd factory
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.729224    2556 factory.go:223] Registration of the systemd container factory successfully
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.729394    2556 factory.go:153] Registering CRI-O factory
May 20 15:32:38.545691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.729405    2556 factory.go:223] Registration of the crio container factory successfully
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.729468    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.729486    2556 factory.go:103] Registering Raw factory
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.729500    2556 manager.go:1196] Started watching for new ooms in manager
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.730023    2556 manager.go:319] Starting recovery of all containers
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.741359    2556 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.742884    2556 manager.go:324] Recovery completed
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.747745    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.749040    2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-233.ec2.internal\" not found" node="ip-10-0-134-233.ec2.internal"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.749495    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.749518    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.749530    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientPID"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.749943    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.749953    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.749970    2556 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.753271    2556 policy_none.go:49] "None policy: Start"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.753284    2556 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.753295    2556 state_mem.go:35] "Initializing new in-memory state store"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.795874    2556 manager.go:341] "Starting Device Plugin manager"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.795901    2556 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.795911    2556 server.go:85] "Starting device plugin registration server"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.796081    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.796090    2556 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.796420    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.796473    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.796479    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.797024    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.797078    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.884904    2556 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.886278    2556 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.886300    2556 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.886319    2556 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.886328    2556 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.886374    2556 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.888743    2556 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.896426    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.897129    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.897151    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.897161    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientPID"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.897180    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-233.ec2.internal"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.905466    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-233.ec2.internal"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.905481    2556 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-233.ec2.internal\": node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:37.916960    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.986837    2556 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal"]
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.986889    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.988322    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.988344    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:32:38.546754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.988353    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientPID"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991018    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991166    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991189    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991692    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991717    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991735    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991743    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientPID"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991720    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.991797    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientPID"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.993529    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.993547    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.994154    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.994184    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:37.994202    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeHasSufficientPID"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.017586    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.019595    2556 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-233.ec2.internal\" not found" node="ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.023374    2556 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-233.ec2.internal\" not found" node="ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.030226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56be31beecb6cce1928e19022161dc49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal\" (UID: \"56be31beecb6cce1928e19022161dc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.030254    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56be31beecb6cce1928e19022161dc49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal\" (UID: \"56be31beecb6cce1928e19022161dc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.030287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1258e990010ed0305514767d059e0701-config\") pod \"kube-apiserver-proxy-ip-10-0-134-233.ec2.internal\" (UID: \"1258e990010ed0305514767d059e0701\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.118553    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.130856    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56be31beecb6cce1928e19022161dc49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal\" (UID: \"56be31beecb6cce1928e19022161dc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.130886    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56be31beecb6cce1928e19022161dc49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal\" (UID: \"56be31beecb6cce1928e19022161dc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.130930    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56be31beecb6cce1928e19022161dc49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal\" (UID: \"56be31beecb6cce1928e19022161dc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.130948    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1258e990010ed0305514767d059e0701-config\") pod \"kube-apiserver-proxy-ip-10-0-134-233.ec2.internal\" (UID: \"1258e990010ed0305514767d059e0701\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.547930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.130971    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1258e990010ed0305514767d059e0701-config\") pod \"kube-apiserver-proxy-ip-10-0-134-233.ec2.internal\" (UID: \"1258e990010ed0305514767d059e0701\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.548631 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.131002    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56be31beecb6cce1928e19022161dc49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal\" (UID: \"56be31beecb6cce1928e19022161dc49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.548631 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.219182    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.548631 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.319625    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.548631 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.323422    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.548631 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.326995    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.548631 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.420172    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.548631 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.520891    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.621387 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.621364    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.629531 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.629515    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 15:32:38.629679 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.629661    2556 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:32:38.629739 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.629676    2556 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:32:38.721775 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:38.721755    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-233.ec2.internal\" not found"
May 20 15:32:38.723954 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.723926    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 15:27:37 +0000 UTC" deadline="2027-11-07 15:33:41.670136101 +0000 UTC"
May 20 15:32:38.724022 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.723954    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12864h1m2.946185959s"
May 20 15:32:38.727910 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.727894    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 15:32:38.727984 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.727940    2556 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:32:38.730268 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.730255    2556 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:32:38.737960 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.737945    2556 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:32:38.759696 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.759679    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r47kq"
May 20 15:32:38.765818 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.765803    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r47kq"
May 20 15:32:38.829422 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.829370    2556 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.840909 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.840895    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:32:38.841701 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.841689    2556 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal"
May 20 15:32:38.850600 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.850582    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:32:38.899754 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:38.899730    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56be31beecb6cce1928e19022161dc49.slice/crio-aa79bc219ce391d249473af4c35a16e9ad467fd545cdc41b9ef2e9e2153b9176 WatchSource:0}: Error finding container aa79bc219ce391d249473af4c35a16e9ad467fd545cdc41b9ef2e9e2153b9176: Status 404 returned error can't find the container with id aa79bc219ce391d249473af4c35a16e9ad467fd545cdc41b9ef2e9e2153b9176
May 20 15:32:38.899925 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:38.899902    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1258e990010ed0305514767d059e0701.slice/crio-21f956c99d744cdb992d5eb8d704d3287303514e50f588c47b09d47a91803e04 WatchSource:0}: Error finding container 21f956c99d744cdb992d5eb8d704d3287303514e50f588c47b09d47a91803e04: Status 404 returned error can't find the container with id 21f956c99d744cdb992d5eb8d704d3287303514e50f588c47b09d47a91803e04
May 20 15:32:38.905202 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:38.905185    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:32:39.455220 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.455188    2556 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:32:39.699992 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.699966    2556 apiserver.go:52] "Watching apiserver"
May 20 15:32:39.708044 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.707986    2556 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 15:32:39.708381 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.708360    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hdrhl","kube-system/konnectivity-agent-8qs26","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv","openshift-cluster-node-tuning-operator/tuned-mpbk4","openshift-image-registry/node-ca-dw68c","openshift-multus/multus-z98x6","kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal","openshift-dns/node-resolver-rs4nk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal","openshift-multus/multus-additional-cni-plugins-9cn9s","openshift-multus/network-metrics-daemon-44pnc","openshift-network-diagnostics/network-check-target-8c6s4","openshift-network-operator/iptables-alerter-tqf9l"]
May 20 15:32:39.713179 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.712766    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.713179 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.712868    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8qs26"
May 20 15:32:39.715049 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.715026    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.715590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.715574    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 15:32:39.715898 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.715872    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 15:32:39.716003 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.715931    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 15:32:39.716003 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.715997    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wkl88\""
May 20 15:32:39.716127 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.715997    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 15:32:39.716127 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.716034    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6th5d\""
May 20 15:32:39.716229 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.716144    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 15:32:39.716511 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.716492    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 15:32:39.716594 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.716527    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 15:32:39.717482 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.717464    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.717663 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.717636    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 15:32:39.717873 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.717857    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 15:32:39.717974 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.717960    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g64qs\""
May 20 15:32:39.718287 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.718214    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 15:32:39.719660 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.719594    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 15:32:39.719760 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.719696    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.720084 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.720061    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 15:32:39.720192 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.720120    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 15:32:39.721711 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.721429    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 15:32:39.721711 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.721503    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-25kvr\""
May 20 15:32:39.721711 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.721579    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 15:32:39.723529 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.722162    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 15:32:39.723529 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.722558    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 15:32:39.723529 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.722639    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 15:32:39.723529 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.722646    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 15:32:39.723529 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.722701    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pmln2\""
May 20 15:32:39.725652 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.725623    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z98x6"
May 20 15:32:39.728043 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.727866    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 15:32:39.728043 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.727917    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:39.728043 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:39.727998    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:39.728043 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.728027    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.728815 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.728797    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d8dw7\""
May 20 15:32:39.730412 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.730271    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 15:32:39.730412 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.730305    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xv7q2\""
May 20 15:32:39.730542 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.730431    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 15:32:39.730542 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.730452    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:39.730542 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:39.730521    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:39.732839 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.732821    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.735011 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.734997    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 15:32:39.735011 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.735012    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gkbwl\""
May 20 15:32:39.735383 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.735368    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 15:32:39.736874 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.736853    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.738937 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.738914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.739030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.738969    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cnibin\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.739030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.738996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-cni-multus\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739002    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 15:32:39.739212 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739004    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 15:32:39.739212 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-systemd\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.739212 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739117    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 15:32:39.739212 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:39.739212 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739179    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739213    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-os-release\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739239    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-cni-netd\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739267    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e273678-1148-4636-a24d-895144217e21-ovn-node-metrics-cert\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739305    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-serviceca\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739339    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrg4q\" (UniqueName: \"kubernetes.io/projected/f95984e7-eb1b-431d-9b72-e34522b8e80f-kube-api-access-jrg4q\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739352    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hs8l6\""
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739365    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-cni-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739399    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f82c6669-7244-4512-aaa5-344d379dd5c5-cni-binary-copy\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739441    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-socket-dir-parent\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739494    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-cnibin\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96a608c3-26ba-4e9a-935b-253eecb12586-konnectivity-ca\") pod \"konnectivity-agent-8qs26\" (UID: \"96a608c3-26ba-4e9a-935b-253eecb12586\") " pod="kube-system/konnectivity-agent-8qs26"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlvjd\" (UniqueName: \"kubernetes.io/projected/c81f7e0c-18a6-4c32-931e-852550317209-kube-api-access-qlvjd\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-slash\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739576    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739591    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5skfc\" (UniqueName: \"kubernetes.io/projected/4e273678-1148-4636-a24d-895144217e21-kube-api-access-5skfc\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-system-cni-dir\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-cni-bin\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739662    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwkl\" (UniqueName: \"kubernetes.io/projected/f82c6669-7244-4512-aaa5-344d379dd5c5-kube-api-access-gdwkl\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739684    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c81f7e0c-18a6-4c32-931e-852550317209-host-slash\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739706    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-run-netns\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739728    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-hostroot\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739753    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c81f7e0c-18a6-4c32-931e-852550317209-iptables-alerter-script\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-ovn\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.739811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739798    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-node-log\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739822    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-env-overrides\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739844    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-host\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739867    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-kubelet\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739890    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-conf-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.739969    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96a608c3-26ba-4e9a-935b-253eecb12586-agent-certs\") pod \"konnectivity-agent-8qs26\" (UID: \"96a608c3-26ba-4e9a-935b-253eecb12586\") " pod="kube-system/konnectivity-agent-8qs26"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740008    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-os-release\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740038    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-netns\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740066    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-ovnkube-config\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740089    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-ovnkube-script-lib\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdjx\" (UniqueName: \"kubernetes.io/projected/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-kube-api-access-nqdjx\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740206    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740234    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-kubelet\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740254    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740271    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-daemon-config\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740291    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-etc-kubernetes\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.740453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740313    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-log-socket\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-cni-bin\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740352    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95984e7-eb1b-431d-9b72-e34522b8e80f-tmp-dir\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f95984e7-eb1b-431d-9b72-e34522b8e80f-hosts-file\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xnd5\" (UniqueName: \"kubernetes.io/projected/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-kube-api-access-2xnd5\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740459    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-multus-certs\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740483    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-systemd-units\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-var-lib-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740519    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740536    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-system-cni-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740559    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-k8s-cni-cncf-io\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740597    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-etc-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740612    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:39.741033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.740649    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghv2j\" (UniqueName: \"kubernetes.io/projected/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-kube-api-access-ghv2j\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:39.766413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.766376    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:27:38 +0000 UTC" deadline="2027-12-24 15:47:23.238650908 +0000 UTC"
May 20 15:32:39.766413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.766401    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13992h14m43.472253803s"
May 20 15:32:39.830065 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.830046    2556 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 15:32:39.840931 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.840903    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-sys-fs\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.841029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.840945    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-cnibin\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.841029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.840973    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96a608c3-26ba-4e9a-935b-253eecb12586-konnectivity-ca\") pod \"konnectivity-agent-8qs26\" (UID: \"96a608c3-26ba-4e9a-935b-253eecb12586\") " pod="kube-system/konnectivity-agent-8qs26"
May 20 15:32:39.841029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.840996    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qlvjd\" (UniqueName: \"kubernetes.io/projected/c81f7e0c-18a6-4c32-931e-852550317209-kube-api-access-qlvjd\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.841029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841021    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-lib-modules\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-host\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841051    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-cnibin\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841063    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eca1dc24-4845-4c65-a1ea-765efbdec276-tmp\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841088    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-slash\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841127    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841151    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5skfc\" (UniqueName: \"kubernetes.io/projected/4e273678-1148-4636-a24d-895144217e21-kube-api-access-5skfc\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r656\" (UniqueName: \"kubernetes.io/projected/c99cddde-b395-4b9a-8724-1de3ce897474-kube-api-access-7r656\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841202    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-system-cni-dir\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841211    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.841237 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841228    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-cni-bin\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841252    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwkl\" (UniqueName: \"kubernetes.io/projected/f82c6669-7244-4512-aaa5-344d379dd5c5-kube-api-access-gdwkl\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841259    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-slash\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841293    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c81f7e0c-18a6-4c32-931e-852550317209-host-slash\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841322    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-system-cni-dir\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841368    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c81f7e0c-18a6-4c32-931e-852550317209-host-slash\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841413    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-cni-bin\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-kubernetes\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysctl-d\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841488    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841537    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-run-netns\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841574    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-modprobe-d\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841599    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-hostroot\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841607    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96a608c3-26ba-4e9a-935b-253eecb12586-konnectivity-ca\") pod \"konnectivity-agent-8qs26\" (UID: \"96a608c3-26ba-4e9a-935b-253eecb12586\") " pod="kube-system/konnectivity-agent-8qs26"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841598    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-run-netns\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841640    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-hostroot\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841625    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c81f7e0c-18a6-4c32-931e-852550317209-iptables-alerter-script\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.841685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841687    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-ovn\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841716    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-node-log\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841732    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-ovn\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841748    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-env-overrides\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841772    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-host\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841793    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-node-log\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841799    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysconfig\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-systemd\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841850    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-host\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841862    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-kubelet\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841895    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-conf-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841907    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-kubelet\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841924    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96a608c3-26ba-4e9a-935b-253eecb12586-agent-certs\") pod \"konnectivity-agent-8qs26\" (UID: \"96a608c3-26ba-4e9a-935b-253eecb12586\") " pod="kube-system/konnectivity-agent-8qs26"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841940    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-conf-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841948    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-tuned\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.841979    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-socket-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-etc-selinux\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842017    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-os-release\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.842417 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842037    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-netns\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842060    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-ovnkube-config\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842082    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-netns\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842087    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-ovnkube-script-lib\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842127    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c81f7e0c-18a6-4c32-931e-852550317209-iptables-alerter-script\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842163    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-os-release\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842158    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdjx\" (UniqueName: \"kubernetes.io/projected/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-kube-api-access-nqdjx\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842194    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-env-overrides\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842207    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842316    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-kubelet\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842362    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-kubelet\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842388    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842409    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842409    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-daemon-config\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842447    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-etc-kubernetes\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842463    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-log-socket\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842478    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-cni-bin\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.843247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842485    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-etc-kubernetes\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842494    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842524    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95984e7-eb1b-431d-9b72-e34522b8e80f-tmp-dir\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842530    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842549    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f95984e7-eb1b-431d-9b72-e34522b8e80f-hosts-file\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842559    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-log-socket\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842563    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-ovnkube-script-lib\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842574    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-run\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842586    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-cni-bin\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842599    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-device-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842622    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f95984e7-eb1b-431d-9b72-e34522b8e80f-hosts-file\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842629    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xnd5\" (UniqueName: \"kubernetes.io/projected/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-kube-api-access-2xnd5\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842654    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-multus-certs\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842668    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e273678-1148-4636-a24d-895144217e21-ovnkube-config\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842677    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-systemd-units\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842700    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-var-lib-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842714    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-multus-certs\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.844006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842726    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842743    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-systemd-units\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842750    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-system-cni-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842775    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-k8s-cni-cncf-io\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842769    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-var-lib-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842801    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-etc-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842806    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842825    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842848    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-system-cni-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842277    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842861    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95984e7-eb1b-431d-9b72-e34522b8e80f-tmp-dir\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842854    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ghv2j\" (UniqueName: \"kubernetes.io/projected/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-kube-api-access-ghv2j\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842886    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-etc-openvswitch\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842896    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysctl-conf\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842921    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-run-k8s-cni-cncf-io\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842922    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-var-lib-kubelet\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842953    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.844834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842953    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842968    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-daemon-config\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.842973    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cnibin\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843005    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-cni-multus\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843026    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cnibin\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843031    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-systemd\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843057    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-run-systemd\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843041    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-host-var-lib-cni-multus\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843070    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843116    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-sys\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843147    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843173    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-os-release\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:39.843193    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843229    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-cni-netd\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843197    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e273678-1148-4636-a24d-895144217e21-host-cni-netd\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843259    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-os-release\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:39.843280    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:40.343242487 +0000 UTC m=+3.034498411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:39.845535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843308    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e273678-1148-4636-a24d-895144217e21-ovn-node-metrics-cert\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843328    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-serviceca\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843343    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jrg4q\" (UniqueName: \"kubernetes.io/projected/f95984e7-eb1b-431d-9b72-e34522b8e80f-kube-api-access-jrg4q\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843365    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-registration-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843378    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843462    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-cni-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843494    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f82c6669-7244-4512-aaa5-344d379dd5c5-cni-binary-copy\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843520    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-socket-dir-parent\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843549    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7km\" (UniqueName: \"kubernetes.io/projected/eca1dc24-4845-4c65-a1ea-765efbdec276-kube-api-access-lz7km\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843639    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-socket-dir-parent\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843639    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f82c6669-7244-4512-aaa5-344d379dd5c5-multus-cni-dir\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843666    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843771    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-serviceca\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.846407 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.843927    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f82c6669-7244-4512-aaa5-344d379dd5c5-cni-binary-copy\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.847021 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.846869    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e273678-1148-4636-a24d-895144217e21-ovn-node-metrics-cert\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.847021 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.846993    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96a608c3-26ba-4e9a-935b-253eecb12586-agent-certs\") pod \"konnectivity-agent-8qs26\" (UID: \"96a608c3-26ba-4e9a-935b-253eecb12586\") " pod="kube-system/konnectivity-agent-8qs26"
May 20 15:32:39.851668 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.851647    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlvjd\" (UniqueName: \"kubernetes.io/projected/c81f7e0c-18a6-4c32-931e-852550317209-kube-api-access-qlvjd\") pod \"iptables-alerter-tqf9l\" (UID: \"c81f7e0c-18a6-4c32-931e-852550317209\") " pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:39.851791 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.851771    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwkl\" (UniqueName: \"kubernetes.io/projected/f82c6669-7244-4512-aaa5-344d379dd5c5-kube-api-access-gdwkl\") pod \"multus-z98x6\" (UID: \"f82c6669-7244-4512-aaa5-344d379dd5c5\") " pod="openshift-multus/multus-z98x6"
May 20 15:32:39.851958 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.851936    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xnd5\" (UniqueName: \"kubernetes.io/projected/72179c2e-fdb0-4e86-86ec-7be1ff47ea67-kube-api-access-2xnd5\") pod \"multus-additional-cni-plugins-9cn9s\" (UID: \"72179c2e-fdb0-4e86-86ec-7be1ff47ea67\") " pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:39.853295 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:39.853265    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:39.853295 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:39.853287    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:39.853295 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:39.853299    2556 projected.go:194] Error preparing data for projected volume kube-api-access-675hr for pod openshift-network-diagnostics/network-check-target-8c6s4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:39.853486 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:39.853350    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr podName:b909f61b-515c-409b-9fa7-8ae2dea1e342 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:40.353338037 +0000 UTC m=+3.044593956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-675hr" (UniqueName: "kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr") pod "network-check-target-8c6s4" (UID: "b909f61b-515c-409b-9fa7-8ae2dea1e342") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:39.855521 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.855497    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghv2j\" (UniqueName: \"kubernetes.io/projected/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-kube-api-access-ghv2j\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:39.855617 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.855586    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdjx\" (UniqueName: \"kubernetes.io/projected/ea0f63ed-dc2b-42b1-8ad6-b83b95c52940-kube-api-access-nqdjx\") pod \"node-ca-dw68c\" (UID: \"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940\") " pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:39.855749 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.855731    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrg4q\" (UniqueName: \"kubernetes.io/projected/f95984e7-eb1b-431d-9b72-e34522b8e80f-kube-api-access-jrg4q\") pod \"node-resolver-rs4nk\" (UID: \"f95984e7-eb1b-431d-9b72-e34522b8e80f\") " pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:39.856286 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.856270    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skfc\" (UniqueName: \"kubernetes.io/projected/4e273678-1148-4636-a24d-895144217e21-kube-api-access-5skfc\") pod \"ovnkube-node-hdrhl\" (UID: \"4e273678-1148-4636-a24d-895144217e21\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:39.891489 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.891449    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal" event={"ID":"1258e990010ed0305514767d059e0701","Type":"ContainerStarted","Data":"21f956c99d744cdb992d5eb8d704d3287303514e50f588c47b09d47a91803e04"}
May 20 15:32:39.892540 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.892515    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal" event={"ID":"56be31beecb6cce1928e19022161dc49","Type":"ContainerStarted","Data":"aa79bc219ce391d249473af4c35a16e9ad467fd545cdc41b9ef2e9e2153b9176"}
May 20 15:32:39.944545 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944522    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-run\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944704 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944556    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-device-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.944704 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944598    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysctl-conf\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944704 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944633    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-var-lib-kubelet\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944704 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944657    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-device-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.944704 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944660    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-run\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944704 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944699    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-sys\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944729    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-registration-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944728    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-var-lib-kubelet\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944759    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7km\" (UniqueName: \"kubernetes.io/projected/eca1dc24-4845-4c65-a1ea-765efbdec276-kube-api-access-lz7km\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944785    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-sys-fs\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944787    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysctl-conf\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944795    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-sys\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944806    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-registration-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944812    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-lib-modules\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944856    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-sys-fs\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944885    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-host\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944909    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eca1dc24-4845-4c65-a1ea-765efbdec276-tmp\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944928    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-host\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944938    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7r656\" (UniqueName: \"kubernetes.io/projected/c99cddde-b395-4b9a-8724-1de3ce897474-kube-api-access-7r656\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.944983 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944941    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-lib-modules\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.944999    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-kubernetes\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945024    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysctl-d\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945154    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysctl-d\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945169    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-kubernetes\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945189    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945208    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-modprobe-d\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945228    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysconfig\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945226    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945272    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-sysconfig\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945278    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-systemd\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945302    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-modprobe-d\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945317    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-tuned\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945347    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-socket-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945391    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-etc-selinux\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945377    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-systemd\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945508    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-socket-dir\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.945640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.945534    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c99cddde-b395-4b9a-8724-1de3ce897474-etc-selinux\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.947567 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.947544    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eca1dc24-4845-4c65-a1ea-765efbdec276-tmp\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.947675 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.947642    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eca1dc24-4845-4c65-a1ea-765efbdec276-etc-tuned\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:39.955848 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.955831    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r656\" (UniqueName: \"kubernetes.io/projected/c99cddde-b395-4b9a-8724-1de3ce897474-kube-api-access-7r656\") pod \"aws-ebs-csi-driver-node-lgkrv\" (UID: \"c99cddde-b395-4b9a-8724-1de3ce897474\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:39.955958 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:39.955933    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7km\" (UniqueName: \"kubernetes.io/projected/eca1dc24-4845-4c65-a1ea-765efbdec276-kube-api-access-lz7km\") pod \"tuned-mpbk4\" (UID: \"eca1dc24-4845-4c65-a1ea-765efbdec276\") " pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:40.026874 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.026807    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9cn9s"
May 20 15:32:40.033718 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.033699    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8qs26"
May 20 15:32:40.042179 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.042159    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tqf9l"
May 20 15:32:40.048948 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.048926    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:32:40.055553 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.055536    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dw68c"
May 20 15:32:40.064134 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.064113    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z98x6"
May 20 15:32:40.070667 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.070649    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rs4nk"
May 20 15:32:40.075242 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.075223    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mpbk4"
May 20 15:32:40.080779 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.080761    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv"
May 20 15:32:40.162149 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.162123    2556 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:32:40.347832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.347757    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:40.347981 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:40.347891    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:40.347981 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:40.347960    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:41.347941844 +0000 UTC m=+4.039197767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:40.448358 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.448329    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:40.448491 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:40.448474    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:40.448555 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:40.448499    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:40.448555 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:40.448511    2556 projected.go:194] Error preparing data for projected volume kube-api-access-675hr for pod openshift-network-diagnostics/network-check-target-8c6s4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:40.448636 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:40.448569    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr podName:b909f61b-515c-409b-9fa7-8ae2dea1e342 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:41.448549865 +0000 UTC m=+4.139805774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-675hr" (UniqueName: "kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr") pod "network-check-target-8c6s4" (UID: "b909f61b-515c-409b-9fa7-8ae2dea1e342") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:40.451305 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:40.451281    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a608c3_26ba_4e9a_935b_253eecb12586.slice/crio-4e411abce07fc3e97471340aef584079985e7f62cc76502b76dc2899b2975ca9 WatchSource:0}: Error finding container 4e411abce07fc3e97471340aef584079985e7f62cc76502b76dc2899b2975ca9: Status 404 returned error can't find the container with id 4e411abce07fc3e97471340aef584079985e7f62cc76502b76dc2899b2975ca9
May 20 15:32:40.452868 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:40.452847    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0f63ed_dc2b_42b1_8ad6_b83b95c52940.slice/crio-b2185d270b5f71ed0787c8874ad496246328945ac0732316cef9df6f01fc39b1 WatchSource:0}: Error finding container b2185d270b5f71ed0787c8874ad496246328945ac0732316cef9df6f01fc39b1: Status 404 returned error can't find the container with id b2185d270b5f71ed0787c8874ad496246328945ac0732316cef9df6f01fc39b1
May 20 15:32:40.453404 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:40.453336    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca1dc24_4845_4c65_a1ea_765efbdec276.slice/crio-20cd773c7b4ee04815204994fde43179b51244f37873e875b271801b989e9348 WatchSource:0}: Error finding container 20cd773c7b4ee04815204994fde43179b51244f37873e875b271801b989e9348: Status 404 returned error can't find the container with id 20cd773c7b4ee04815204994fde43179b51244f37873e875b271801b989e9348
May 20 15:32:40.454299 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:40.454231    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81f7e0c_18a6_4c32_931e_852550317209.slice/crio-41a38ab5f3cc0a626d81b499c548d6e31f7056b5b75a7b6c69822cbfaa7b68ee WatchSource:0}: Error finding container 41a38ab5f3cc0a626d81b499c548d6e31f7056b5b75a7b6c69822cbfaa7b68ee: Status 404 returned error can't find the container with id 41a38ab5f3cc0a626d81b499c548d6e31f7056b5b75a7b6c69822cbfaa7b68ee
May 20 15:32:40.457010 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:40.456348    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e273678_1148_4636_a24d_895144217e21.slice/crio-37f452fd58f44ea1d04c620970cc876b88e6d9e5ec5baf67bdabb19ba31b6e92 WatchSource:0}: Error finding container 37f452fd58f44ea1d04c620970cc876b88e6d9e5ec5baf67bdabb19ba31b6e92: Status 404 returned error can't find the container with id 37f452fd58f44ea1d04c620970cc876b88e6d9e5ec5baf67bdabb19ba31b6e92
May 20 15:32:40.457010 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:40.456752    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72179c2e_fdb0_4e86_86ec_7be1ff47ea67.slice/crio-d9c40958c82408e802afe80d40d9bcef1b69931af4a0713d9fda0f5ca7baccb4 WatchSource:0}: Error finding container d9c40958c82408e802afe80d40d9bcef1b69931af4a0713d9fda0f5ca7baccb4: Status 404 returned error can't find the container with id d9c40958c82408e802afe80d40d9bcef1b69931af4a0713d9fda0f5ca7baccb4
May 20 15:32:40.462684 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:40.462646    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82c6669_7244_4512_aaa5_344d379dd5c5.slice/crio-07223d316ea4aaa68736d27daec0b79a8df23c92bcbe16fabde54a14d780b710 WatchSource:0}: Error finding container 07223d316ea4aaa68736d27daec0b79a8df23c92bcbe16fabde54a14d780b710: Status 404 returned error can't find the container with id 07223d316ea4aaa68736d27daec0b79a8df23c92bcbe16fabde54a14d780b710
May 20 15:32:40.486582 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:32:40.486563    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99cddde_b395_4b9a_8724_1de3ce897474.slice/crio-4c7739d14d5fe0f140c74ec05455159e909de7eb2d6c9313f4245ef563911709 WatchSource:0}: Error finding container 4c7739d14d5fe0f140c74ec05455159e909de7eb2d6c9313f4245ef563911709: Status 404 returned error can't find the container with id 4c7739d14d5fe0f140c74ec05455159e909de7eb2d6c9313f4245ef563911709
May 20 15:32:40.766987 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.766844    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:27:38 +0000 UTC" deadline="2028-02-29 20:26:10.898582086 +0000 UTC"
May 20 15:32:40.767335 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.766988    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15604h53m30.131598289s"
May 20 15:32:40.888032 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.887558    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:40.888032 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:40.887681    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:40.899514 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.899446    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tqf9l" event={"ID":"c81f7e0c-18a6-4c32-931e-852550317209","Type":"ContainerStarted","Data":"41a38ab5f3cc0a626d81b499c548d6e31f7056b5b75a7b6c69822cbfaa7b68ee"}
May 20 15:32:40.901695 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.901668    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mpbk4" event={"ID":"eca1dc24-4845-4c65-a1ea-765efbdec276","Type":"ContainerStarted","Data":"20cd773c7b4ee04815204994fde43179b51244f37873e875b271801b989e9348"}
May 20 15:32:40.903475 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.903451    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8qs26" event={"ID":"96a608c3-26ba-4e9a-935b-253eecb12586","Type":"ContainerStarted","Data":"4e411abce07fc3e97471340aef584079985e7f62cc76502b76dc2899b2975ca9"}
May 20 15:32:40.906935 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.906906    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal" event={"ID":"1258e990010ed0305514767d059e0701","Type":"ContainerStarted","Data":"b4fdf31f85837eee6a66724789a3b9e5738670207754c868fe98e14daedd0473"}
May 20 15:32:40.910004 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.909982    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z98x6" event={"ID":"f82c6669-7244-4512-aaa5-344d379dd5c5","Type":"ContainerStarted","Data":"07223d316ea4aaa68736d27daec0b79a8df23c92bcbe16fabde54a14d780b710"}
May 20 15:32:40.912342 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.912315    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerStarted","Data":"d9c40958c82408e802afe80d40d9bcef1b69931af4a0713d9fda0f5ca7baccb4"}
May 20 15:32:40.914326 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.914302    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"37f452fd58f44ea1d04c620970cc876b88e6d9e5ec5baf67bdabb19ba31b6e92"}
May 20 15:32:40.917615 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.917592    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dw68c" event={"ID":"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940","Type":"ContainerStarted","Data":"b2185d270b5f71ed0787c8874ad496246328945ac0732316cef9df6f01fc39b1"}
May 20 15:32:40.920429 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.920403    2556 generic.go:358] "Generic (PLEG): container finished" podID="56be31beecb6cce1928e19022161dc49" containerID="652b0d6b468415ce5af489b8a1c2d2da1a9fa9d29d36046b6ee0f0da52eef2f9" exitCode=0
May 20 15:32:40.920518 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.920471    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal" event={"ID":"56be31beecb6cce1928e19022161dc49","Type":"ContainerDied","Data":"652b0d6b468415ce5af489b8a1c2d2da1a9fa9d29d36046b6ee0f0da52eef2f9"}
May 20 15:32:40.921324 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.921273    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-233.ec2.internal" podStartSLOduration=2.921259606 podStartE2EDuration="2.921259606s" podCreationTimestamp="2026-05-20 15:32:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:40.92106365 +0000 UTC m=+3.612319583" watchObservedRunningTime="2026-05-20 15:32:40.921259606 +0000 UTC m=+3.612515517"
May 20 15:32:40.928180 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.928160    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv" event={"ID":"c99cddde-b395-4b9a-8724-1de3ce897474","Type":"ContainerStarted","Data":"4c7739d14d5fe0f140c74ec05455159e909de7eb2d6c9313f4245ef563911709"}
May 20 15:32:40.929999 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:40.929975    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rs4nk" event={"ID":"f95984e7-eb1b-431d-9b72-e34522b8e80f","Type":"ContainerStarted","Data":"9bc55f8384da37b39fada1ce8c3d2f759a328fd05358632f90fbfa082972ecbe"}
May 20 15:32:41.354227 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:41.354145    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:41.354381 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:41.354287    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:41.354381 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:41.354347    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:43.35432992 +0000 UTC m=+6.045585846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:41.454832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:41.454798    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:41.455005 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:41.454984    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:41.455005 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:41.455004    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:41.455160 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:41.455017    2556 projected.go:194] Error preparing data for projected volume kube-api-access-675hr for pod openshift-network-diagnostics/network-check-target-8c6s4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:41.455160 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:41.455071    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr podName:b909f61b-515c-409b-9fa7-8ae2dea1e342 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:43.455054824 +0000 UTC m=+6.146310732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-675hr" (UniqueName: "kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr") pod "network-check-target-8c6s4" (UID: "b909f61b-515c-409b-9fa7-8ae2dea1e342") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:41.889127 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:41.888887    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:41.889127 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:41.889018    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:41.940994 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:41.940967    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal" event={"ID":"56be31beecb6cce1928e19022161dc49","Type":"ContainerStarted","Data":"91bb6ae584d225d25050d2523538bdcf8cc4d58c4a968389eb8686bb1df5eaa5"}
May 20 15:32:42.887117 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:42.887075    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:42.887286 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:42.887204    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:43.368514 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:43.368433    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:43.368935 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:43.368571    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:43.368935 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:43.368639    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:47.368618785 +0000 UTC m=+10.059874709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:43.469089 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:43.469058    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:43.469260 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:43.469220    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:43.469260 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:43.469238    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:43.469260 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:43.469249    2556 projected.go:194] Error preparing data for projected volume kube-api-access-675hr for pod openshift-network-diagnostics/network-check-target-8c6s4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:43.469434 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:43.469304    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr podName:b909f61b-515c-409b-9fa7-8ae2dea1e342 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:47.469285517 +0000 UTC m=+10.160541442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-675hr" (UniqueName: "kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr") pod "network-check-target-8c6s4" (UID: "b909f61b-515c-409b-9fa7-8ae2dea1e342") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:43.887624 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:43.887172    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:43.887624 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:43.887313    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:44.886636 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:44.886602    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:44.887061 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:44.886724    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:45.887473 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:45.887443    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:45.887903 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:45.887569    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:46.887419 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:46.887391    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:46.887592 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:46.887513    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:47.400844 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:47.400812    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:47.401012 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:47.400969    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:47.401075 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:47.401030    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:55.40101149 +0000 UTC m=+18.092267402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:47.501494 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:47.501451    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:47.501657 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:47.501631    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:47.501657 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:47.501649    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:47.501775 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:47.501662    2556 projected.go:194] Error preparing data for projected volume kube-api-access-675hr for pod openshift-network-diagnostics/network-check-target-8c6s4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:47.501775 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:47.501718    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr podName:b909f61b-515c-409b-9fa7-8ae2dea1e342 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:55.501703175 +0000 UTC m=+18.192959085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-675hr" (UniqueName: "kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr") pod "network-check-target-8c6s4" (UID: "b909f61b-515c-409b-9fa7-8ae2dea1e342") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:47.887348 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:47.887264    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:47.888688 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:47.888647    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:48.887157 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:48.887127    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:48.887331 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:48.887239    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:49.591358 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.591306    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-233.ec2.internal" podStartSLOduration=11.591286702 podStartE2EDuration="11.591286702s" podCreationTimestamp="2026-05-20 15:32:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:41.966899733 +0000 UTC m=+4.658155662" watchObservedRunningTime="2026-05-20 15:32:49.591286702 +0000 UTC m=+12.282542631"
May 20 15:32:49.592448 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.591872    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-9lt8j"]
May 20 15:32:49.604646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.604624    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.604762 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:49.604711    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:32:49.718538 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.718505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.718697 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.718550    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45de9667-3479-4085-8ff9-bbc7b04205c2-dbus\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.718697 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.718646    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45de9667-3479-4085-8ff9-bbc7b04205c2-kubelet-config\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.819357 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.819328    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.819357 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.819360    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45de9667-3479-4085-8ff9-bbc7b04205c2-dbus\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.819528 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:49.819472    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:32:49.819528 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.819490    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45de9667-3479-4085-8ff9-bbc7b04205c2-kubelet-config\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.819528 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.819522    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45de9667-3479-4085-8ff9-bbc7b04205c2-dbus\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.819654 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:49.819528    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret podName:45de9667-3479-4085-8ff9-bbc7b04205c2 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:50.319510782 +0000 UTC m=+13.010766706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret") pod "global-pull-secret-syncer-9lt8j" (UID: "45de9667-3479-4085-8ff9-bbc7b04205c2") : object "kube-system"/"original-pull-secret" not registered
May 20 15:32:49.819654 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.819584    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45de9667-3479-4085-8ff9-bbc7b04205c2-kubelet-config\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:49.890268 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:49.889802    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:49.890268 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:49.889921    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:50.324030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:50.323967    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:50.324168 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:50.324055    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:32:50.324168 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:50.324113    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret podName:45de9667-3479-4085-8ff9-bbc7b04205c2 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:51.324087691 +0000 UTC m=+14.015343597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret") pod "global-pull-secret-syncer-9lt8j" (UID: "45de9667-3479-4085-8ff9-bbc7b04205c2") : object "kube-system"/"original-pull-secret" not registered
May 20 15:32:50.886693 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:50.886666    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:50.886693 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:50.886682    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:50.887199 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:50.886772    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:32:50.887199 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:50.886845    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:51.331653 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:51.331577    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:51.331788 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:51.331697    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:32:51.331788 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:51.331747    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret podName:45de9667-3479-4085-8ff9-bbc7b04205c2 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:53.331732466 +0000 UTC m=+16.022988371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret") pod "global-pull-secret-syncer-9lt8j" (UID: "45de9667-3479-4085-8ff9-bbc7b04205c2") : object "kube-system"/"original-pull-secret" not registered
May 20 15:32:51.886978 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:51.886950    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:51.887452 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:51.887086    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:52.887419 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:52.887390    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:52.887860 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:52.887390    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:52.887860 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:52.887515    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:32:52.887860 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:52.887593    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:53.347441 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:53.347408    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:53.347593 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:53.347551    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:32:53.347663 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:53.347617    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret podName:45de9667-3479-4085-8ff9-bbc7b04205c2 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:57.347598004 +0000 UTC m=+20.038853938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret") pod "global-pull-secret-syncer-9lt8j" (UID: "45de9667-3479-4085-8ff9-bbc7b04205c2") : object "kube-system"/"original-pull-secret" not registered
May 20 15:32:53.889706 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:53.889677    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:53.890200 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:53.889795    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:54.887145 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:54.887117    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:54.887336 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:54.887117    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:54.887336 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:54.887226    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:54.887336 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:54.887316    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:32:55.466223 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:55.466190    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:55.466624 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:55.466298    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:55.466624 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:55.466358    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:11.46633958 +0000 UTC m=+34.157595486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:55.566948 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:55.566911    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:55.567126 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:55.567089    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:55.567126 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:55.567127    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:55.567310 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:55.567141    2556 projected.go:194] Error preparing data for projected volume kube-api-access-675hr for pod openshift-network-diagnostics/network-check-target-8c6s4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:55.567310 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:55.567196    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr podName:b909f61b-515c-409b-9fa7-8ae2dea1e342 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:11.567178023 +0000 UTC m=+34.258433983 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-675hr" (UniqueName: "kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr") pod "network-check-target-8c6s4" (UID: "b909f61b-515c-409b-9fa7-8ae2dea1e342") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:55.889296 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:55.889265    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:55.889469 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:55.889398    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:56.886982 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:56.886950    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:56.887409 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:56.887066    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:32:56.887409 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:56.887127    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:56.887409 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:56.887232    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:57.384294 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:57.384247    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:57.384549 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:57.384372    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:32:57.384549 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:57.384439    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret podName:45de9667-3479-4085-8ff9-bbc7b04205c2 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:05.384419058 +0000 UTC m=+28.075674977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret") pod "global-pull-secret-syncer-9lt8j" (UID: "45de9667-3479-4085-8ff9-bbc7b04205c2") : object "kube-system"/"original-pull-secret" not registered
May 20 15:32:57.890908 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:57.890869    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:57.891342 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:57.891260    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:58.886964 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.886669    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:32:58.886964 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.886683    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:32:58.887122 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:58.886978    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:32:58.887122 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:58.887028    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:32:58.967515 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.967479    2556 generic.go:358] "Generic (PLEG): container finished" podID="72179c2e-fdb0-4e86-86ec-7be1ff47ea67" containerID="d338ddbbe5e665d3a4d0f36ba4125ba751f53df44083beebdba59c50cd7ecca3" exitCode=0
May 20 15:32:58.968326 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.967574    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerDied","Data":"d338ddbbe5e665d3a4d0f36ba4125ba751f53df44083beebdba59c50cd7ecca3"}
May 20 15:32:58.969424 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.969403    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"97018ebb5a0aa90c014d8378d854b1397443b9882fb98ea5722c6955d1e8bc72"}
May 20 15:32:58.969514 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.969429    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"66f11b62d10c475fd6f1b1688b010cf74eabbd3f07f961ea80b1b840a316d052"}
May 20 15:32:58.970671 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.970647    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dw68c" event={"ID":"ea0f63ed-dc2b-42b1-8ad6-b83b95c52940","Type":"ContainerStarted","Data":"37a9466f1c88c4406fc39ec56f7e93d93b4a4329c074c5027c4ce742fab757a2"}
May 20 15:32:58.974701 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.974676    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv" event={"ID":"c99cddde-b395-4b9a-8724-1de3ce897474","Type":"ContainerStarted","Data":"8068988f04ae59d60458dde773b03ab89fc05495c09c1bac9139dcc238c173da"}
May 20 15:32:58.975829 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.975802    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rs4nk" event={"ID":"f95984e7-eb1b-431d-9b72-e34522b8e80f","Type":"ContainerStarted","Data":"cdaa5339bbe085e7cbecb1ef10fc6ac38e652c5bdcb19a1b2a135502fa4cefa4"}
May 20 15:32:58.976928 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.976910    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mpbk4" event={"ID":"eca1dc24-4845-4c65-a1ea-765efbdec276","Type":"ContainerStarted","Data":"22ee207aecef604490d96f2c506af377876290e4703f1f969a18b165e0956110"}
May 20 15:32:58.978093 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.978072    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8qs26" event={"ID":"96a608c3-26ba-4e9a-935b-253eecb12586","Type":"ContainerStarted","Data":"ff79ca75de733a68e5f918df624d630100fad5e200dbbc5dd4b056bf48f155d0"}
May 20 15:32:58.979167 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:58.979150    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z98x6" event={"ID":"f82c6669-7244-4512-aaa5-344d379dd5c5","Type":"ContainerStarted","Data":"b70864c336f8b8476a41399498cbfde346df01d17baae21dc7ca86ee6c7c1c6b"}
May 20 15:32:59.014750 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.014704    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-rs4nk" podStartSLOduration=3.979337784 podStartE2EDuration="22.014690368s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.485042257 +0000 UTC m=+3.176298164" lastFinishedPulling="2026-05-20 15:32:58.520394842 +0000 UTC m=+21.211650748" observedRunningTime="2026-05-20 15:32:59.014627789 +0000 UTC m=+21.705883717" watchObservedRunningTime="2026-05-20 15:32:59.014690368 +0000 UTC m=+21.705946296"
May 20 15:32:59.031382 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.031344    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8qs26" podStartSLOduration=12.030993528 podStartE2EDuration="22.031334168s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.453043461 +0000 UTC m=+3.144299368" lastFinishedPulling="2026-05-20 15:32:50.453384101 +0000 UTC m=+13.144640008" observedRunningTime="2026-05-20 15:32:59.030980205 +0000 UTC m=+21.722236144" watchObservedRunningTime="2026-05-20 15:32:59.031334168 +0000 UTC m=+21.722590095"
May 20 15:32:59.051820 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.051791    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dw68c" podStartSLOduration=3.985497346 podStartE2EDuration="22.051782318s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.454770039 +0000 UTC m=+3.146025962" lastFinishedPulling="2026-05-20 15:32:58.521055025 +0000 UTC m=+21.212310934" observedRunningTime="2026-05-20 15:32:59.051269902 +0000 UTC m=+21.742525831" watchObservedRunningTime="2026-05-20 15:32:59.051782318 +0000 UTC m=+21.743038246"
May 20 15:32:59.081397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.081365    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-z98x6" podStartSLOduration=4.015825237 podStartE2EDuration="22.081355861s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.485024738 +0000 UTC m=+3.176280644" lastFinishedPulling="2026-05-20 15:32:58.550555362 +0000 UTC m=+21.241811268" observedRunningTime="2026-05-20 15:32:59.073216595 +0000 UTC m=+21.764472548" watchObservedRunningTime="2026-05-20 15:32:59.081355861 +0000 UTC m=+21.772611790"
May 20 15:32:59.092946 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.092913    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mpbk4" podStartSLOduration=3.025534901 podStartE2EDuration="21.092905591s" podCreationTimestamp="2026-05-20 15:32:38 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.455137496 +0000 UTC m=+3.146393404" lastFinishedPulling="2026-05-20 15:32:58.522508174 +0000 UTC m=+21.213764094" observedRunningTime="2026-05-20 15:32:59.092519646 +0000 UTC m=+21.783775573" watchObservedRunningTime="2026-05-20 15:32:59.092905591 +0000 UTC m=+21.784161519"
May 20 15:32:59.887318 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.887250    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:32:59.887455 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:32:59.887372    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:32:59.982674 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.982647    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tqf9l" event={"ID":"c81f7e0c-18a6-4c32-931e-852550317209","Type":"ContainerStarted","Data":"10bd5bd2ca792cd50084f1b58760370f0ce731fabed5e890451fbdc4950e8363"}
May 20 15:32:59.986003 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.985973    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"bce485495dd497340853579e22368714e86ea08aea117e049349f7c411a1db00"}
May 20 15:32:59.986844 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.986010    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"0805f001d0165bf31f23cc3e405bad3f44f545b4445553146df091ca2dccce52"}
May 20 15:32:59.986844 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.986023    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"522fe19bd0dd0b4ccf7274b3c416e4ad83e02093b1e1c694ac199ed7cd72ee3b"}
May 20 15:32:59.986844 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:32:59.986035    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"508fc21c3ba64483da5cb81a44f50c34d59eb60e9215cbe11b4b27815f74df9d"}
May 20 15:33:00.004223 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.004177    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tqf9l" podStartSLOduration=4.960655253 podStartE2EDuration="23.004162837s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.457951077 +0000 UTC m=+3.149206994" lastFinishedPulling="2026-05-20 15:32:58.501458659 +0000 UTC m=+21.192714578" observedRunningTime="2026-05-20 15:33:00.004060918 +0000 UTC m=+22.695316847" watchObservedRunningTime="2026-05-20 15:33:00.004162837 +0000 UTC m=+22.695418765"
May 20 15:33:00.086351 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.086328    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 15:33:00.294274 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.294244    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8qs26"
May 20 15:33:00.294885 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.294868    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8qs26"
May 20 15:33:00.810427 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.810194    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T15:33:00.086348582Z","UUID":"869d2cbd-9922-465b-9287-eda0a13dbe9b","Handler":null,"Name":"","Endpoint":""}
May 20 15:33:00.811965 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.811934    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 15:33:00.811965 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.811967    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 15:33:00.887468 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.887447    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:00.887586 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.887447    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:00.887630 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:00.887535    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:33:00.887630 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:00.887614    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:33:00.988768 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.988739    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv" event={"ID":"c99cddde-b395-4b9a-8724-1de3ce897474","Type":"ContainerStarted","Data":"27b7b59cd1d24608230bac0c7366fc10e9a59d162e82cb17fea2e8b9bca65aec"}
May 20 15:33:00.989236 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.989019    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8qs26"
May 20 15:33:00.989378 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:00.989362    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8qs26"
May 20 15:33:01.887495 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:01.887465    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:01.887675 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:01.887580    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:33:01.993770 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:01.993737    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"54c268145f83f54d116f26b3ae8c6628137540248d2d6a93b49ce146f08ee894"}
May 20 15:33:01.995827 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:01.995785    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv" event={"ID":"c99cddde-b395-4b9a-8724-1de3ce897474","Type":"ContainerStarted","Data":"e3e29386ac9011da34cb726834efd6de02e42768d862b5ce9d909a488d234bb2"}
May 20 15:33:02.018358 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:02.018309    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lgkrv" podStartSLOduration=3.387407674 podStartE2EDuration="24.018296413s" podCreationTimestamp="2026-05-20 15:32:38 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.488522586 +0000 UTC m=+3.179778492" lastFinishedPulling="2026-05-20 15:33:01.119411322 +0000 UTC m=+23.810667231" observedRunningTime="2026-05-20 15:33:02.017743329 +0000 UTC m=+24.708999259" watchObservedRunningTime="2026-05-20 15:33:02.018296413 +0000 UTC m=+24.709552340"
May 20 15:33:02.886519 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:02.886489    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:02.886683 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:02.886488    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:02.886683 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:02.886617    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:33:02.886683 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:02.886648    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:33:03.886614 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:03.886579    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:03.887340 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:03.886714    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:33:04.000150 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.000119    2556 generic.go:358] "Generic (PLEG): container finished" podID="72179c2e-fdb0-4e86-86ec-7be1ff47ea67" containerID="2821b8ec1621be7ba498416981ca3925e14dde3c28212b199f23fb5c2cab8e64" exitCode=0
May 20 15:33:04.000256 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.000195    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerDied","Data":"2821b8ec1621be7ba498416981ca3925e14dde3c28212b199f23fb5c2cab8e64"}
May 20 15:33:04.003354 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.003304    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" event={"ID":"4e273678-1148-4636-a24d-895144217e21","Type":"ContainerStarted","Data":"2faafa45f23d0252722c22ddb18a643b657527915e980c2901cfecab1e3284e3"}
May 20 15:33:04.003598 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.003580    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:33:04.003680 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.003608    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:33:04.018976 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.018859    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:33:04.019070 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.019025    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:33:04.072140 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.072086    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl" podStartSLOduration=8.80664012 podStartE2EDuration="27.072075454s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.458783278 +0000 UTC m=+3.150039184" lastFinishedPulling="2026-05-20 15:32:58.72421861 +0000 UTC m=+21.415474518" observedRunningTime="2026-05-20 15:33:04.071778241 +0000 UTC m=+26.763034169" watchObservedRunningTime="2026-05-20 15:33:04.072075454 +0000 UTC m=+26.763331403"
May 20 15:33:04.375278 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.375184    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:33:04.886547 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.886518    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:04.886722 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:04.886518    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:04.886722 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:04.886627    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:33:04.887054 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:04.886727    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:33:05.447534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:05.447499    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:05.447675 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:05.447653    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:33:05.447727 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:05.447716    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret podName:45de9667-3479-4085-8ff9-bbc7b04205c2 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:21.447701992 +0000 UTC m=+44.138957898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret") pod "global-pull-secret-syncer-9lt8j" (UID: "45de9667-3479-4085-8ff9-bbc7b04205c2") : object "kube-system"/"original-pull-secret" not registered
May 20 15:33:05.591508 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:05.591475    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8c6s4"]
May 20 15:33:05.591658 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:05.591588    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:05.591694 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:05.591665    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:33:05.594334 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:05.594304    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-44pnc"]
May 20 15:33:05.594457 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:05.594443    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:05.594578 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:05.594556    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:33:05.594956 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:05.594933    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9lt8j"]
May 20 15:33:05.595070 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:05.595047    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:05.595201 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:05.595173    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:33:06.008142 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:06.008112    2556 generic.go:358] "Generic (PLEG): container finished" podID="72179c2e-fdb0-4e86-86ec-7be1ff47ea67" containerID="72a8e7458add3d8ce3f08be8835560f4cabe4b05bdef5e9b8c5e95d30b6929b9" exitCode=0
May 20 15:33:06.008142 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:06.008125    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerDied","Data":"72a8e7458add3d8ce3f08be8835560f4cabe4b05bdef5e9b8c5e95d30b6929b9"}
May 20 15:33:06.886748 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:06.886676    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:06.886859 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:06.886681    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:06.886859 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:06.886776    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:33:06.886859 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:06.886799    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:06.886998 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:06.886912    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:33:06.887031 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:06.887009    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:33:07.011954 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:07.011925    2556 generic.go:358] "Generic (PLEG): container finished" podID="72179c2e-fdb0-4e86-86ec-7be1ff47ea67" containerID="1143a3d240fda89e05481be0ed7526940de28a5605f0f4ad7e681019841d3a9d" exitCode=0
May 20 15:33:07.012288 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:07.011997    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerDied","Data":"1143a3d240fda89e05481be0ed7526940de28a5605f0f4ad7e681019841d3a9d"}
May 20 15:33:08.887031 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:08.886816    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:08.887446 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:08.886857    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:08.887446 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:08.887153    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:33:08.887446 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:08.886880    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:08.887446 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:08.887198    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:33:08.887446 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:08.887280    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:33:10.887480 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:10.887446    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:10.888286 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:10.887446    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:10.888286 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:10.887446    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:10.888286 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:10.888087    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:33:10.888499 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:10.888467    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9lt8j" podUID="45de9667-3479-4085-8ff9-bbc7b04205c2"
May 20 15:33:10.888638 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:10.888591    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c6s4" podUID="b909f61b-515c-409b-9fa7-8ae2dea1e342"
May 20 15:33:11.491217 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:11.491177    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:11.491417 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:11.491309    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:33:11.491417 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:11.491384    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:43.491362109 +0000 UTC m=+66.182618017 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:33:11.592435 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:11.592402    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:11.592597 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:11.592578    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:33:11.592644 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:11.592600    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:33:11.592644 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:11.592610    2556 projected.go:194] Error preparing data for projected volume kube-api-access-675hr for pod openshift-network-diagnostics/network-check-target-8c6s4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:33:11.592710 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:11.592662    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr podName:b909f61b-515c-409b-9fa7-8ae2dea1e342 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:43.592648766 +0000 UTC m=+66.283904677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-675hr" (UniqueName: "kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr") pod "network-check-target-8c6s4" (UID: "b909f61b-515c-409b-9fa7-8ae2dea1e342") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:33:12.614697 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.614672    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-233.ec2.internal" event="NodeReady"
May 20 15:33:12.615140 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.614778    2556 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 15:33:12.678561 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.678537    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4z29q"]
May 20 15:33:12.701045 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.701007    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b9xjr"]
May 20 15:33:12.701226 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.701197    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.704211 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.704189    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j6j2b\""
May 20 15:33:12.704410 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.704393    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 15:33:12.711666 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.711647    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 15:33:12.721244 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.721147    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9xjr"]
May 20 15:33:12.721244 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.721178    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4z29q"]
May 20 15:33:12.721413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.721302    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:12.723748 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.723729    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 15:33:12.723748 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.723740    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 15:33:12.723902 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.723750    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 15:33:12.723902 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.723789    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-94zvt\""
May 20 15:33:12.802691 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.802658    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bbd465-a5f9-43a2-b969-0f6be7230e22-tmp-dir\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.802869 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.802708    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:12.802869 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.802746    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbwc\" (UniqueName: \"kubernetes.io/projected/a46d22fb-89c2-4738-bdce-892fb1a3fd53-kube-api-access-kgbwc\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:12.802869 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.802773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xdpx\" (UniqueName: \"kubernetes.io/projected/b9bbd465-a5f9-43a2-b969-0f6be7230e22-kube-api-access-7xdpx\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.802869 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.802805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.802869 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.802834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bbd465-a5f9-43a2-b969-0f6be7230e22-config-volume\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.886878 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.886854    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:12.886958 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.886865    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:12.887295 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.887078    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:12.890361 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.890008    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:33:12.890361 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.890067    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xw7hl\""
May 20 15:33:12.890361 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.890082    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:33:12.890361 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.890067    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-79qfs\""
May 20 15:33:12.890361 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.890127    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 15:33:12.890361 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.890146    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:33:12.903706 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.903684    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.903782 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.903727    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bbd465-a5f9-43a2-b969-0f6be7230e22-config-volume\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.903841 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:12.903804    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:33:12.903889 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:12.903857    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls podName:b9bbd465-a5f9-43a2-b969-0f6be7230e22 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:13.403840659 +0000 UTC m=+36.095096565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls") pod "dns-default-4z29q" (UID: "b9bbd465-a5f9-43a2-b969-0f6be7230e22") : secret "dns-default-metrics-tls" not found
May 20 15:33:12.903889 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.903801    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bbd465-a5f9-43a2-b969-0f6be7230e22-tmp-dir\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.903998 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.903900    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:12.903998 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.903919    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbwc\" (UniqueName: \"kubernetes.io/projected/a46d22fb-89c2-4738-bdce-892fb1a3fd53-kube-api-access-kgbwc\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:12.903998 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.903940    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdpx\" (UniqueName: \"kubernetes.io/projected/b9bbd465-a5f9-43a2-b969-0f6be7230e22-kube-api-access-7xdpx\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.904180 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:12.904039    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:33:12.904180 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:12.904146    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert podName:a46d22fb-89c2-4738-bdce-892fb1a3fd53 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:13.404132053 +0000 UTC m=+36.095387962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert") pod "ingress-canary-b9xjr" (UID: "a46d22fb-89c2-4738-bdce-892fb1a3fd53") : secret "canary-serving-cert" not found
May 20 15:33:12.904286 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.904243    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bbd465-a5f9-43a2-b969-0f6be7230e22-tmp-dir\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.904405 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.904387    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bbd465-a5f9-43a2-b969-0f6be7230e22-config-volume\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.917816 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.917793    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdpx\" (UniqueName: \"kubernetes.io/projected/b9bbd465-a5f9-43a2-b969-0f6be7230e22-kube-api-access-7xdpx\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:12.917907 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:12.917897    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbwc\" (UniqueName: \"kubernetes.io/projected/a46d22fb-89c2-4738-bdce-892fb1a3fd53-kube-api-access-kgbwc\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:13.026625 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:13.026593    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerStarted","Data":"b49f8b72ed1ac1c4134fdf4f17859cace194c904cca4efd8c209c40b85e7de80"}
May 20 15:33:13.407036 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:13.406977    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:13.407159 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:13.407066    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:13.407159 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:13.407138    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:33:13.407246 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:13.407167    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:33:13.407246 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:13.407209    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls podName:b9bbd465-a5f9-43a2-b969-0f6be7230e22 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:14.407190404 +0000 UTC m=+37.098446324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls") pod "dns-default-4z29q" (UID: "b9bbd465-a5f9-43a2-b969-0f6be7230e22") : secret "dns-default-metrics-tls" not found
May 20 15:33:13.407246 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:13.407224    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert podName:a46d22fb-89c2-4738-bdce-892fb1a3fd53 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:14.40721702 +0000 UTC m=+37.098472926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert") pod "ingress-canary-b9xjr" (UID: "a46d22fb-89c2-4738-bdce-892fb1a3fd53") : secret "canary-serving-cert" not found
May 20 15:33:14.030331 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:14.030300    2556 generic.go:358] "Generic (PLEG): container finished" podID="72179c2e-fdb0-4e86-86ec-7be1ff47ea67" containerID="b49f8b72ed1ac1c4134fdf4f17859cace194c904cca4efd8c209c40b85e7de80" exitCode=0
May 20 15:33:14.030909 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:14.030362    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerDied","Data":"b49f8b72ed1ac1c4134fdf4f17859cace194c904cca4efd8c209c40b85e7de80"}
May 20 15:33:14.414380 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:14.414309    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:14.414380 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:14.414361    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:14.414567 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:14.414465    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:33:14.414567 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:14.414511    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:33:14.414567 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:14.414528    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert podName:a46d22fb-89c2-4738-bdce-892fb1a3fd53 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:16.414511807 +0000 UTC m=+39.105767732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert") pod "ingress-canary-b9xjr" (UID: "a46d22fb-89c2-4738-bdce-892fb1a3fd53") : secret "canary-serving-cert" not found
May 20 15:33:14.414567 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:14.414562    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls podName:b9bbd465-a5f9-43a2-b969-0f6be7230e22 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:16.414545838 +0000 UTC m=+39.105801764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls") pod "dns-default-4z29q" (UID: "b9bbd465-a5f9-43a2-b969-0f6be7230e22") : secret "dns-default-metrics-tls" not found
May 20 15:33:15.034363 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:15.034328    2556 generic.go:358] "Generic (PLEG): container finished" podID="72179c2e-fdb0-4e86-86ec-7be1ff47ea67" containerID="3fde5282c979184cd7777fdee868bd0a5584523fc57cea4f3ca77d93fac97c4e" exitCode=0
May 20 15:33:15.034802 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:15.034391    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerDied","Data":"3fde5282c979184cd7777fdee868bd0a5584523fc57cea4f3ca77d93fac97c4e"}
May 20 15:33:16.039275 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:16.039234    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" event={"ID":"72179c2e-fdb0-4e86-86ec-7be1ff47ea67","Type":"ContainerStarted","Data":"e8ed4e64452fb798a8df6895974a3b33f364fe339dca9afaadfdbbe7cef0683f"}
May 20 15:33:16.090172 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:16.090117    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9cn9s" podStartSLOduration=6.7209261 podStartE2EDuration="39.090082484s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:32:40.462476901 +0000 UTC m=+3.153732823" lastFinishedPulling="2026-05-20 15:33:12.831633301 +0000 UTC m=+35.522889207" observedRunningTime="2026-05-20 15:33:16.088908749 +0000 UTC m=+38.780164700" watchObservedRunningTime="2026-05-20 15:33:16.090082484 +0000 UTC m=+38.781338412"
May 20 15:33:16.430505 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:16.430432    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:16.430505 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:16.430504    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:16.430686 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:16.430561    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:33:16.430686 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:16.430594    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:33:16.430686 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:16.430620    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls podName:b9bbd465-a5f9-43a2-b969-0f6be7230e22 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:20.430604495 +0000 UTC m=+43.121860400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls") pod "dns-default-4z29q" (UID: "b9bbd465-a5f9-43a2-b969-0f6be7230e22") : secret "dns-default-metrics-tls" not found
May 20 15:33:16.430686 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:16.430635    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert podName:a46d22fb-89c2-4738-bdce-892fb1a3fd53 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:20.430628471 +0000 UTC m=+43.121884376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert") pod "ingress-canary-b9xjr" (UID: "a46d22fb-89c2-4738-bdce-892fb1a3fd53") : secret "canary-serving-cert" not found
May 20 15:33:20.458628 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:20.458581    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:20.459017 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:20.458637    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:20.459017 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:20.458745    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:33:20.459017 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:20.458745    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:33:20.459017 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:20.458813    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert podName:a46d22fb-89c2-4738-bdce-892fb1a3fd53 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:28.458794625 +0000 UTC m=+51.150050539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert") pod "ingress-canary-b9xjr" (UID: "a46d22fb-89c2-4738-bdce-892fb1a3fd53") : secret "canary-serving-cert" not found
May 20 15:33:20.459017 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:20.458830    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls podName:b9bbd465-a5f9-43a2-b969-0f6be7230e22 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:28.458821867 +0000 UTC m=+51.150077773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls") pod "dns-default-4z29q" (UID: "b9bbd465-a5f9-43a2-b969-0f6be7230e22") : secret "dns-default-metrics-tls" not found
May 20 15:33:21.465520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:21.465489    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:21.468543 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:21.468513    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45de9667-3479-4085-8ff9-bbc7b04205c2-original-pull-secret\") pod \"global-pull-secret-syncer-9lt8j\" (UID: \"45de9667-3479-4085-8ff9-bbc7b04205c2\") " pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:21.599464 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:21.599440    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9lt8j"
May 20 15:33:21.777382 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:21.777346    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9lt8j"]
May 20 15:33:21.781333 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:33:21.781302    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45de9667_3479_4085_8ff9_bbc7b04205c2.slice/crio-19e5952e70335b250482d13a0f0e82952d533a89a7526118485b0bed17524f48 WatchSource:0}: Error finding container 19e5952e70335b250482d13a0f0e82952d533a89a7526118485b0bed17524f48: Status 404 returned error can't find the container with id 19e5952e70335b250482d13a0f0e82952d533a89a7526118485b0bed17524f48
May 20 15:33:22.051298 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.051258    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9lt8j" event={"ID":"45de9667-3479-4085-8ff9-bbc7b04205c2","Type":"ContainerStarted","Data":"19e5952e70335b250482d13a0f0e82952d533a89a7526118485b0bed17524f48"}
May 20 15:33:22.901482 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.901445    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"]
May 20 15:33:22.919351 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.919326    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"]
May 20 15:33:22.919501 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.919456    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:22.921974 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.921948    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 20 15:33:22.922128 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.921954    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 20 15:33:22.923197 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.923180    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 20 15:33:22.923197 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.923190    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 20 15:33:22.952951 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.952895    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"]
May 20 15:33:22.975245 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.975226    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"]
May 20 15:33:22.975346 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.975322    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:22.978133 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.977991    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 20 15:33:22.978133 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.978043    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 20 15:33:22.978133 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.978078    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 20 15:33:22.978133 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:22.978078    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 20 15:33:23.076368 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-klusterlet-config\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.076534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-ca\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.076534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076404    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.076534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076428    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.076534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076447    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hkm\" (UniqueName: \"kubernetes.io/projected/4c113c03-3b96-4e91-843d-7c7ce941e8bd-kube-api-access-m7hkm\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.076534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076476    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/4c113c03-3b96-4e91-843d-7c7ce941e8bd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.076764 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076565    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-hub\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.076764 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076596    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-tmp\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.076764 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.076693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrx6v\" (UniqueName: \"kubernetes.io/projected/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-kube-api-access-xrx6v\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.177678 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177601    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xrx6v\" (UniqueName: \"kubernetes.io/projected/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-kube-api-access-xrx6v\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.177832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177679    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-klusterlet-config\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.177832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177722    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-ca\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.177832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177746    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.177832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177776    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.177832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177801    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hkm\" (UniqueName: \"kubernetes.io/projected/4c113c03-3b96-4e91-843d-7c7ce941e8bd-kube-api-access-m7hkm\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.178078 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177839    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/4c113c03-3b96-4e91-843d-7c7ce941e8bd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.178078 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177874    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-hub\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.178078 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.177898    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-tmp\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.178298 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.178276    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-tmp\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.178953 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.178884    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/4c113c03-3b96-4e91-843d-7c7ce941e8bd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.180815 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.180795    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-ca\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.181313 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.181284    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.181420 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.181345    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-hub\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.181462 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.181434    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-klusterlet-config\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.181740 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.181717    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c113c03-3b96-4e91-843d-7c7ce941e8bd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.190978 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.190950    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hkm\" (UniqueName: \"kubernetes.io/projected/4c113c03-3b96-4e91-843d-7c7ce941e8bd-kube-api-access-m7hkm\") pod \"cluster-proxy-proxy-agent-784d9ff8ff-68vzl\" (UID: \"4c113c03-3b96-4e91-843d-7c7ce941e8bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.191093 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.190993    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrx6v\" (UniqueName: \"kubernetes.io/projected/7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e-kube-api-access-xrx6v\") pod \"klusterlet-addon-workmgr-f4bf7dc4d-7cpzr\" (UID: \"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.231592 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.231570    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:23.293633 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.293602    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:33:23.388375 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.388345    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"]
May 20 15:33:23.392665 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:33:23.392238    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0aad07_6d96_4b91_bcbb_9ce0a53a3e3e.slice/crio-7a049e95f7bcbd4574003eb4d806a5e15e65b6a0fc8c6d55bf2d391b408d80b0 WatchSource:0}: Error finding container 7a049e95f7bcbd4574003eb4d806a5e15e65b6a0fc8c6d55bf2d391b408d80b0: Status 404 returned error can't find the container with id 7a049e95f7bcbd4574003eb4d806a5e15e65b6a0fc8c6d55bf2d391b408d80b0
May 20 15:33:23.445731 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:23.445650    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"]
May 20 15:33:23.448996 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:33:23.448971    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c113c03_3b96_4e91_843d_7c7ce941e8bd.slice/crio-6bf656993e971f24d99054de9f57a71d76f4c56e42fd9a3e09a7d3845ec27fa7 WatchSource:0}: Error finding container 6bf656993e971f24d99054de9f57a71d76f4c56e42fd9a3e09a7d3845ec27fa7: Status 404 returned error can't find the container with id 6bf656993e971f24d99054de9f57a71d76f4c56e42fd9a3e09a7d3845ec27fa7
May 20 15:33:24.058554 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:24.058503    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" event={"ID":"4c113c03-3b96-4e91-843d-7c7ce941e8bd","Type":"ContainerStarted","Data":"6bf656993e971f24d99054de9f57a71d76f4c56e42fd9a3e09a7d3845ec27fa7"}
May 20 15:33:24.060248 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:24.060199    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr" event={"ID":"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e","Type":"ContainerStarted","Data":"7a049e95f7bcbd4574003eb4d806a5e15e65b6a0fc8c6d55bf2d391b408d80b0"}
May 20 15:33:28.516963 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:28.516923    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:28.517420 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:28.516970    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:28.517420 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:28.517068    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:33:28.517420 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:28.517078    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:33:28.517420 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:28.517148    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls podName:b9bbd465-a5f9-43a2-b969-0f6be7230e22 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:44.517129588 +0000 UTC m=+67.208385501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls") pod "dns-default-4z29q" (UID: "b9bbd465-a5f9-43a2-b969-0f6be7230e22") : secret "dns-default-metrics-tls" not found
May 20 15:33:28.517420 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:28.517162    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert podName:a46d22fb-89c2-4738-bdce-892fb1a3fd53 nodeName:}" failed. No retries permitted until 2026-05-20 15:33:44.517156405 +0000 UTC m=+67.208412310 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert") pod "ingress-canary-b9xjr" (UID: "a46d22fb-89c2-4738-bdce-892fb1a3fd53") : secret "canary-serving-cert" not found
May 20 15:33:30.075325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:30.075205    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9lt8j" event={"ID":"45de9667-3479-4085-8ff9-bbc7b04205c2","Type":"ContainerStarted","Data":"feaa18008bbd267c3fc7966e4b0a8afc206da4671bb18d0410bb32c06b4431b4"}
May 20 15:33:30.076714 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:30.076681    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" event={"ID":"4c113c03-3b96-4e91-843d-7c7ce941e8bd","Type":"ContainerStarted","Data":"b88b019787fcbbd0e58335ac7dfddc11a3990a8c66c4dafdb02cb2224d5d30c6"}
May 20 15:33:30.078037 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:30.077997    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr" event={"ID":"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e","Type":"ContainerStarted","Data":"9c00719c94f0dc160eb0d150c595cfec4fd2e142d928c7c1f1ba9bdcb1caf4f1"}
May 20 15:33:30.078289 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:30.078261    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:30.079338 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:30.079312    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:33:30.123646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:30.123608    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9lt8j" podStartSLOduration=32.977822431 podStartE2EDuration="41.123596497s" podCreationTimestamp="2026-05-20 15:32:49 +0000 UTC" firstStartedPulling="2026-05-20 15:33:21.78314465 +0000 UTC m=+44.474400559" lastFinishedPulling="2026-05-20 15:33:29.928918704 +0000 UTC m=+52.620174625" observedRunningTime="2026-05-20 15:33:30.104175894 +0000 UTC m=+52.795431839" watchObservedRunningTime="2026-05-20 15:33:30.123596497 +0000 UTC m=+52.814852424"
May 20 15:33:30.123834 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:30.123815    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr" podStartSLOduration=1.569021378 podStartE2EDuration="8.123810166s" podCreationTimestamp="2026-05-20 15:33:22 +0000 UTC" firstStartedPulling="2026-05-20 15:33:23.394719653 +0000 UTC m=+46.085975560" lastFinishedPulling="2026-05-20 15:33:29.949508428 +0000 UTC m=+52.640764348" observedRunningTime="2026-05-20 15:33:30.123095122 +0000 UTC m=+52.814351052" watchObservedRunningTime="2026-05-20 15:33:30.123810166 +0000 UTC m=+52.815066135"
May 20 15:33:32.084054 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:32.084028    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" event={"ID":"4c113c03-3b96-4e91-843d-7c7ce941e8bd","Type":"ContainerStarted","Data":"7be0b02bcb5b99dbfa955baeca62b7bc9806365fd219812c8e34eb020930df95"}
May 20 15:33:32.084334 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:32.084068    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" event={"ID":"4c113c03-3b96-4e91-843d-7c7ce941e8bd","Type":"ContainerStarted","Data":"a8a7ccefcec317b7da8f808af6677c939b2ffbdb9bc023b031a490276ada92cf"}
May 20 15:33:32.122807 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:32.122768    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" podStartSLOduration=1.6212299369999998 podStartE2EDuration="10.122755695s" podCreationTimestamp="2026-05-20 15:33:22 +0000 UTC" firstStartedPulling="2026-05-20 15:33:23.450929774 +0000 UTC m=+46.142185680" lastFinishedPulling="2026-05-20 15:33:31.952455531 +0000 UTC m=+54.643711438" observedRunningTime="2026-05-20 15:33:32.12103186 +0000 UTC m=+54.812287788" watchObservedRunningTime="2026-05-20 15:33:32.122755695 +0000 UTC m=+54.814011623"
May 20 15:33:36.023413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:36.023377    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdrhl"
May 20 15:33:43.522587 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.522551    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:33:43.525755 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.525735    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:33:43.532927 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:43.532907    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 15:33:43.532989 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:43.532969    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:47.532950687 +0000 UTC m=+130.224206593 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : secret "metrics-daemon-secret" not found
May 20 15:33:43.622864 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.622830    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:43.626040 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.626022    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:33:43.636213 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.636192    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:33:43.646865 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.646844    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-675hr\" (UniqueName: \"kubernetes.io/projected/b909f61b-515c-409b-9fa7-8ae2dea1e342-kube-api-access-675hr\") pod \"network-check-target-8c6s4\" (UID: \"b909f61b-515c-409b-9fa7-8ae2dea1e342\") " pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:43.809591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.809528    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xw7hl\""
May 20 15:33:43.816888 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.816869    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:43.939364 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:43.939338    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8c6s4"]
May 20 15:33:43.942278 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:33:43.942255    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb909f61b_515c_409b_9fa7_8ae2dea1e342.slice/crio-65d57703514fcfe9d5183557526835dae9e6663c61234dcc125043fafc6fa0b3 WatchSource:0}: Error finding container 65d57703514fcfe9d5183557526835dae9e6663c61234dcc125043fafc6fa0b3: Status 404 returned error can't find the container with id 65d57703514fcfe9d5183557526835dae9e6663c61234dcc125043fafc6fa0b3
May 20 15:33:44.105940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:44.105873    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8c6s4" event={"ID":"b909f61b-515c-409b-9fa7-8ae2dea1e342","Type":"ContainerStarted","Data":"65d57703514fcfe9d5183557526835dae9e6663c61234dcc125043fafc6fa0b3"}
May 20 15:33:44.528192 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:44.528161    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:33:44.528563 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:44.528232    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:33:44.528563 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:44.528319    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:33:44.528563 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:44.528346    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:33:44.528563 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:44.528391    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls podName:b9bbd465-a5f9-43a2-b969-0f6be7230e22 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:16.528370031 +0000 UTC m=+99.219625950 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls") pod "dns-default-4z29q" (UID: "b9bbd465-a5f9-43a2-b969-0f6be7230e22") : secret "dns-default-metrics-tls" not found
May 20 15:33:44.528563 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:33:44.528410    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert podName:a46d22fb-89c2-4738-bdce-892fb1a3fd53 nodeName:}" failed. No retries permitted until 2026-05-20 15:34:16.528400622 +0000 UTC m=+99.219656542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert") pod "ingress-canary-b9xjr" (UID: "a46d22fb-89c2-4738-bdce-892fb1a3fd53") : secret "canary-serving-cert" not found
May 20 15:33:47.113669 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:47.113604    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8c6s4" event={"ID":"b909f61b-515c-409b-9fa7-8ae2dea1e342","Type":"ContainerStarted","Data":"6a306a9a07e2dbb55c0455402a60cc4b04aa89e424952ddb5c4bcef6ce6c8e8f"}
May 20 15:33:47.114001 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:47.113761    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:33:47.132291 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:33:47.132248    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8c6s4" podStartSLOduration=67.248100396 podStartE2EDuration="1m10.132235562s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:33:43.943991431 +0000 UTC m=+66.635247336" lastFinishedPulling="2026-05-20 15:33:46.828126592 +0000 UTC m=+69.519382502" observedRunningTime="2026-05-20 15:33:47.132001426 +0000 UTC m=+69.823257354" watchObservedRunningTime="2026-05-20 15:33:47.132235562 +0000 UTC m=+69.823491483"
May 20 15:34:16.541370 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:34:16.541319    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:34:16.541797 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:34:16.541381    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:34:16.541797 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:34:16.541469    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:34:16.541797 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:34:16.541489    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:34:16.541797 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:34:16.541556    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert podName:a46d22fb-89c2-4738-bdce-892fb1a3fd53 nodeName:}" failed. No retries permitted until 2026-05-20 15:35:20.541536159 +0000 UTC m=+163.232792065 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert") pod "ingress-canary-b9xjr" (UID: "a46d22fb-89c2-4738-bdce-892fb1a3fd53") : secret "canary-serving-cert" not found
May 20 15:34:16.541797 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:34:16.541573    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls podName:b9bbd465-a5f9-43a2-b969-0f6be7230e22 nodeName:}" failed. No retries permitted until 2026-05-20 15:35:20.541567032 +0000 UTC m=+163.232822937 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls") pod "dns-default-4z29q" (UID: "b9bbd465-a5f9-43a2-b969-0f6be7230e22") : secret "dns-default-metrics-tls" not found
May 20 15:34:18.118117 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:34:18.118072    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8c6s4"
May 20 15:34:47.550961 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:34:47.550920    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:34:47.551552 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:34:47.551034    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 15:34:47.551552 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:34:47.551121    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs podName:e67e163d-ec7d-4d41-b2b6-27e8bbab3825 nodeName:}" failed. No retries permitted until 2026-05-20 15:36:49.551085529 +0000 UTC m=+252.242341435 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs") pod "network-metrics-daemon-44pnc" (UID: "e67e163d-ec7d-4d41-b2b6-27e8bbab3825") : secret "metrics-daemon-secret" not found
May 20 15:34:59.845575 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:34:59.845547    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rs4nk_f95984e7-eb1b-431d-9b72-e34522b8e80f/dns-node-resolver/0.log"
May 20 15:35:00.443769 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:00.443742    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dw68c_ea0f63ed-dc2b-42b1-8ad6-b83b95c52940/node-ca/0.log"
May 20 15:35:15.710472 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:35:15.710436    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4z29q" podUID="b9bbd465-a5f9-43a2-b969-0f6be7230e22"
May 20 15:35:15.729591 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:35:15.729557    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-b9xjr" podUID="a46d22fb-89c2-4738-bdce-892fb1a3fd53"
May 20 15:35:15.944813 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:35:15.944774    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-44pnc" podUID="e67e163d-ec7d-4d41-b2b6-27e8bbab3825"
May 20 15:35:16.323260 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:16.323231    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4z29q"
May 20 15:35:20.567338 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:20.567300    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:35:20.567906 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:20.567385    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:35:20.569569 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:20.569547    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bbd465-a5f9-43a2-b969-0f6be7230e22-metrics-tls\") pod \"dns-default-4z29q\" (UID: \"b9bbd465-a5f9-43a2-b969-0f6be7230e22\") " pod="openshift-dns/dns-default-4z29q"
May 20 15:35:20.569682 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:20.569664    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46d22fb-89c2-4738-bdce-892fb1a3fd53-cert\") pod \"ingress-canary-b9xjr\" (UID: \"a46d22fb-89c2-4738-bdce-892fb1a3fd53\") " pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:35:20.826789 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:20.826708    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j6j2b\""
May 20 15:35:20.834900 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:20.834879    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4z29q"
May 20 15:35:20.959408 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:20.959379    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4z29q"]
May 20 15:35:20.962832 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:35:20.962803    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bbd465_a5f9_43a2_b969_0f6be7230e22.slice/crio-77ce41e5e62bcc8e75d3e8d851877bbc667d6c010d876acfa00a8173162f0ec7 WatchSource:0}: Error finding container 77ce41e5e62bcc8e75d3e8d851877bbc667d6c010d876acfa00a8173162f0ec7: Status 404 returned error can't find the container with id 77ce41e5e62bcc8e75d3e8d851877bbc667d6c010d876acfa00a8173162f0ec7
May 20 15:35:21.334492 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:21.334463    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4z29q" event={"ID":"b9bbd465-a5f9-43a2-b969-0f6be7230e22","Type":"ContainerStarted","Data":"77ce41e5e62bcc8e75d3e8d851877bbc667d6c010d876acfa00a8173162f0ec7"}
May 20 15:35:23.080360 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.080322    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zbm2j"]
May 20 15:35:23.083263 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.083247    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.087176 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.086984    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6wcqs\""
May 20 15:35:23.087176 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.087006    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 15:35:23.087176 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.087021    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 15:35:23.087176 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.087041    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 15:35:23.087176 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.087005    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 15:35:23.096655 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.096636    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zbm2j"]
May 20 15:35:23.166984 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.166956    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67948b89b7-mm82h"]
May 20 15:35:23.169676 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.169662    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.172012 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.171994    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2gccf\""
May 20 15:35:23.172348 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.172330    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 20 15:35:23.172471 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.172447    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 20 15:35:23.172560 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.172447    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 20 15:35:23.176998 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.176980    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 20 15:35:23.180721 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.180703    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67948b89b7-mm82h"]
May 20 15:35:23.186149 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.186130    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9c9219-d33c-4426-8aa2-d340b4cc6559-crio-socket\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.186227 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.186177    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9c9219-d33c-4426-8aa2-d340b4cc6559-data-volume\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.186227 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.186196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9c9219-d33c-4426-8aa2-d340b4cc6559-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.186296 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.186245    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9c9219-d33c-4426-8aa2-d340b4cc6559-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.186296 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.186282    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmwv\" (UniqueName: \"kubernetes.io/projected/df9c9219-d33c-4426-8aa2-d340b4cc6559-kube-api-access-bjmwv\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.287037 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287016    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-bound-sa-token\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.287182 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287044    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a35d922-49ce-401d-85e3-cafb1b11f072-ca-trust-extracted\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.287182 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287063    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a35d922-49ce-401d-85e3-cafb1b11f072-installation-pull-secrets\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.287182 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287086    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9c9219-d33c-4426-8aa2-d340b4cc6559-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.287182 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287148    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a35d922-49ce-401d-85e3-cafb1b11f072-image-registry-private-configuration\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.287389 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287271    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9c9219-d33c-4426-8aa2-d340b4cc6559-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.287389 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287313    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a35d922-49ce-401d-85e3-cafb1b11f072-trusted-ca\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.287389 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287348    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9c9219-d33c-4426-8aa2-d340b4cc6559-crio-socket\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.287389 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287376    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9vn\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-kube-api-access-2f9vn\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.287562 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287414    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9c9219-d33c-4426-8aa2-d340b4cc6559-crio-socket\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.287562 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287424    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-registry-tls\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.287562 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287495    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9c9219-d33c-4426-8aa2-d340b4cc6559-data-volume\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.287562 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287540    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmwv\" (UniqueName: \"kubernetes.io/projected/df9c9219-d33c-4426-8aa2-d340b4cc6559-kube-api-access-bjmwv\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.287755 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287572    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a35d922-49ce-401d-85e3-cafb1b11f072-registry-certificates\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.287806 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287761    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9c9219-d33c-4426-8aa2-d340b4cc6559-data-volume\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.287852 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.287825    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9c9219-d33c-4426-8aa2-d340b4cc6559-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.289508 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.289487    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9c9219-d33c-4426-8aa2-d340b4cc6559-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.298805 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.298786    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmwv\" (UniqueName: \"kubernetes.io/projected/df9c9219-d33c-4426-8aa2-d340b4cc6559-kube-api-access-bjmwv\") pod \"insights-runtime-extractor-zbm2j\" (UID: \"df9c9219-d33c-4426-8aa2-d340b4cc6559\") " pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.343263 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.343196    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4z29q" event={"ID":"b9bbd465-a5f9-43a2-b969-0f6be7230e22","Type":"ContainerStarted","Data":"9084472a0cd74c21144861cca2fe858361c557eb354a2553d34d30fda961d7c5"}
May 20 15:35:23.343263 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.343231    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4z29q" event={"ID":"b9bbd465-a5f9-43a2-b969-0f6be7230e22","Type":"ContainerStarted","Data":"ff6414de69b75a5f9d7cf4607dd1d15e35fb1da5ca7b139cdea30dda8be66387"}
May 20 15:35:23.343390 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.343376    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4z29q"
May 20 15:35:23.363530 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.363484    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-4z29q" podStartSLOduration=129.960109502 podStartE2EDuration="2m11.363461758s" podCreationTimestamp="2026-05-20 15:33:12 +0000 UTC" firstStartedPulling="2026-05-20 15:35:20.964497975 +0000 UTC m=+163.655753884" lastFinishedPulling="2026-05-20 15:35:22.367850231 +0000 UTC m=+165.059106140" observedRunningTime="2026-05-20 15:35:23.362452476 +0000 UTC m=+166.053708405" watchObservedRunningTime="2026-05-20 15:35:23.363461758 +0000 UTC m=+166.054717687"
May 20 15:35:23.388077 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388048    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-bound-sa-token\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.388077 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388076    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a35d922-49ce-401d-85e3-cafb1b11f072-ca-trust-extracted\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.388256 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388115    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a35d922-49ce-401d-85e3-cafb1b11f072-installation-pull-secrets\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.388256 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388184    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a35d922-49ce-401d-85e3-cafb1b11f072-image-registry-private-configuration\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.388256 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388222    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a35d922-49ce-401d-85e3-cafb1b11f072-trusted-ca\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.388396 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388256    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9vn\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-kube-api-access-2f9vn\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.388396 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388300    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-registry-tls\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.388396 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388346    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a35d922-49ce-401d-85e3-cafb1b11f072-registry-certificates\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.388528 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.388495    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a35d922-49ce-401d-85e3-cafb1b11f072-ca-trust-extracted\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.389254 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.389231    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a35d922-49ce-401d-85e3-cafb1b11f072-registry-certificates\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.389493 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.389450    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a35d922-49ce-401d-85e3-cafb1b11f072-trusted-ca\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.391091 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.390684    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a35d922-49ce-401d-85e3-cafb1b11f072-installation-pull-secrets\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.391091 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.391051    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a35d922-49ce-401d-85e3-cafb1b11f072-image-registry-private-configuration\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.391246 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.391195    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-registry-tls\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.391863 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.391846    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zbm2j"
May 20 15:35:23.395957 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.395941    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-bound-sa-token\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.400226 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.400203    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9vn\" (UniqueName: \"kubernetes.io/projected/4a35d922-49ce-401d-85e3-cafb1b11f072-kube-api-access-2f9vn\") pod \"image-registry-67948b89b7-mm82h\" (UID: \"4a35d922-49ce-401d-85e3-cafb1b11f072\") " pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.478431 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.478407    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:23.533862 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.533832    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zbm2j"]
May 20 15:35:23.536928 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:35:23.536887    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9c9219_d33c_4426_8aa2_d340b4cc6559.slice/crio-98069451f4f462d71e84836a57d4eb3011facf936f686b7a15f3471e1791a71f WatchSource:0}: Error finding container 98069451f4f462d71e84836a57d4eb3011facf936f686b7a15f3471e1791a71f: Status 404 returned error can't find the container with id 98069451f4f462d71e84836a57d4eb3011facf936f686b7a15f3471e1791a71f
May 20 15:35:23.624244 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:23.624216    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67948b89b7-mm82h"]
May 20 15:35:23.628419 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:35:23.628397    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a35d922_49ce_401d_85e3_cafb1b11f072.slice/crio-f402cb338d36e1bbd2a63381d36b0598128082e983e2090572b9b1b5722b9540 WatchSource:0}: Error finding container f402cb338d36e1bbd2a63381d36b0598128082e983e2090572b9b1b5722b9540: Status 404 returned error can't find the container with id f402cb338d36e1bbd2a63381d36b0598128082e983e2090572b9b1b5722b9540
May 20 15:35:24.347143 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:24.347113    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67948b89b7-mm82h" event={"ID":"4a35d922-49ce-401d-85e3-cafb1b11f072","Type":"ContainerStarted","Data":"bf8ad8eee16363089fae5341172dcb5e5b750717cb8354e5815efbbe3a817ad4"}
May 20 15:35:24.347538 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:24.347151    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67948b89b7-mm82h" event={"ID":"4a35d922-49ce-401d-85e3-cafb1b11f072","Type":"ContainerStarted","Data":"f402cb338d36e1bbd2a63381d36b0598128082e983e2090572b9b1b5722b9540"}
May 20 15:35:24.347538 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:24.347188    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:24.348614 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:24.348592    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbm2j" event={"ID":"df9c9219-d33c-4426-8aa2-d340b4cc6559","Type":"ContainerStarted","Data":"d99a9178bd20e63ad7bddbff1536a71d3b8d041842b96e2a5caf28abc915fdad"}
May 20 15:35:24.348709 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:24.348619    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbm2j" event={"ID":"df9c9219-d33c-4426-8aa2-d340b4cc6559","Type":"ContainerStarted","Data":"53db727e88661927818f07f4d14c5855ea8f264560f9d5ee2dd12f12b615af93"}
May 20 15:35:24.348709 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:24.348629    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbm2j" event={"ID":"df9c9219-d33c-4426-8aa2-d340b4cc6559","Type":"ContainerStarted","Data":"98069451f4f462d71e84836a57d4eb3011facf936f686b7a15f3471e1791a71f"}
May 20 15:35:24.370904 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:24.370867    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67948b89b7-mm82h" podStartSLOduration=1.370855847 podStartE2EDuration="1.370855847s" podCreationTimestamp="2026-05-20 15:35:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:35:24.370793779 +0000 UTC m=+167.062049707" watchObservedRunningTime="2026-05-20 15:35:24.370855847 +0000 UTC m=+167.062111774"
May 20 15:35:26.355390 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:26.355313    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbm2j" event={"ID":"df9c9219-d33c-4426-8aa2-d340b4cc6559","Type":"ContainerStarted","Data":"057d89867f2735a596030a4c7928d0dc4c28898c64c1b13f96cf8c3be1d57602"}
May 20 15:35:26.375285 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:26.375244    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zbm2j" podStartSLOduration=1.569545677 podStartE2EDuration="3.375232005s" podCreationTimestamp="2026-05-20 15:35:23 +0000 UTC" firstStartedPulling="2026-05-20 15:35:23.607562415 +0000 UTC m=+166.298818322" lastFinishedPulling="2026-05-20 15:35:25.41324873 +0000 UTC m=+168.104504650" observedRunningTime="2026-05-20 15:35:26.37425963 +0000 UTC m=+169.065515569" watchObservedRunningTime="2026-05-20 15:35:26.375232005 +0000 UTC m=+169.066487963"
May 20 15:35:26.886913 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:26.886887    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:35:26.889689 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:26.889675    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-94zvt\""
May 20 15:35:26.898031 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:26.898009    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9xjr"
May 20 15:35:27.022590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:27.022563    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9xjr"]
May 20 15:35:27.025744 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:35:27.025719    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46d22fb_89c2_4738_bdce_892fb1a3fd53.slice/crio-8cdb096bca9038c13e20abe25c3c307dcb3f8dd47558de53bb7d97af2f0ba5c9 WatchSource:0}: Error finding container 8cdb096bca9038c13e20abe25c3c307dcb3f8dd47558de53bb7d97af2f0ba5c9: Status 404 returned error can't find the container with id 8cdb096bca9038c13e20abe25c3c307dcb3f8dd47558de53bb7d97af2f0ba5c9
May 20 15:35:27.358400 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:27.358372    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9xjr" event={"ID":"a46d22fb-89c2-4738-bdce-892fb1a3fd53","Type":"ContainerStarted","Data":"8cdb096bca9038c13e20abe25c3c307dcb3f8dd47558de53bb7d97af2f0ba5c9"}
May 20 15:35:29.365288 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:29.365231    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9xjr" event={"ID":"a46d22fb-89c2-4738-bdce-892fb1a3fd53","Type":"ContainerStarted","Data":"101fe79f4ba5363e3f8c7a4eeb7d4989176d9669d2124ecf9fbcfc3fb97ba274"}
May 20 15:35:29.382636 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:29.382595    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b9xjr" podStartSLOduration=135.917196219 podStartE2EDuration="2m17.382583103s" podCreationTimestamp="2026-05-20 15:33:12 +0000 UTC" firstStartedPulling="2026-05-20 15:35:27.027574374 +0000 UTC m=+169.718830284" lastFinishedPulling="2026-05-20 15:35:28.492961263 +0000 UTC m=+171.184217168" observedRunningTime="2026-05-20 15:35:29.381876958 +0000 UTC m=+172.073132885" watchObservedRunningTime="2026-05-20 15:35:29.382583103 +0000 UTC m=+172.073839030"
May 20 15:35:29.889910 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:29.889882    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:35:30.079209 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:30.079171    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr" podUID="7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
May 20 15:35:30.369471 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:30.369439    2556 generic.go:358] "Generic (PLEG): container finished" podID="7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e" containerID="9c00719c94f0dc160eb0d150c595cfec4fd2e142d928c7c1f1ba9bdcb1caf4f1" exitCode=1
May 20 15:35:30.369916 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:30.369519    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr" event={"ID":"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e","Type":"ContainerDied","Data":"9c00719c94f0dc160eb0d150c595cfec4fd2e142d928c7c1f1ba9bdcb1caf4f1"}
May 20 15:35:30.369916 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:30.369908    2556 scope.go:117] "RemoveContainer" containerID="9c00719c94f0dc160eb0d150c595cfec4fd2e142d928c7c1f1ba9bdcb1caf4f1"
May 20 15:35:31.374594 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:31.374552    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr" event={"ID":"7c0aad07-6d96-4b91-bcbb-9ce0a53a3e3e","Type":"ContainerStarted","Data":"1b9b6bfb22178d37f6b1c72b1a8863e9d0c09a46692e3e600ba5919630762ae5"}
May 20 15:35:31.375141 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:31.374908    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:35:31.375507 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:31.375488    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f4bf7dc4d-7cpzr"
May 20 15:35:33.351165 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:33.351135    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4z29q"
May 20 15:35:35.524392 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.524355    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kfsrc"]
May 20 15:35:35.527694 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.527670    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.530153 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.530130    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 15:35:35.530329 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.530239    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ttrgj\""
May 20 15:35:35.530448 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.530423    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 15:35:35.531353 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.531337    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 15:35:35.531479 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.531342    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 15:35:35.531638 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.531613    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 15:35:35.531745 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.531655    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 15:35:35.681234 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681208    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-wtmp\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.681341 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-textfile\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.681341 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681259    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31a3434f-9205-4ec2-bb67-e0259aeb1a85-metrics-client-ca\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.681341 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681278    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.681461 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681366    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-tls\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.681461 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681395    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-root\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.681461 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681429    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5568p\" (UniqueName: \"kubernetes.io/projected/31a3434f-9205-4ec2-bb67-e0259aeb1a85-kube-api-access-5568p\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.681549 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681480    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-sys\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.681549 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.681521    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.782695 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782633    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-sys\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.782695 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782681    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.782823 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782722    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-wtmp\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.782823 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782769    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-sys\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.782933 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782831    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-textfile\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.782933 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782868    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-wtmp\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.782933 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782911    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31a3434f-9205-4ec2-bb67-e0259aeb1a85-metrics-client-ca\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.783072 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782933    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.783072 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782967    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-tls\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.783072 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.782989    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-root\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.783072 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.783024    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5568p\" (UniqueName: \"kubernetes.io/projected/31a3434f-9205-4ec2-bb67-e0259aeb1a85-kube-api-access-5568p\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.783299 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.783241    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-textfile\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.783299 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.783248    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a3434f-9205-4ec2-bb67-e0259aeb1a85-root\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.783395 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.783378    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.783537 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.783517    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31a3434f-9205-4ec2-bb67-e0259aeb1a85-metrics-client-ca\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.785681 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.785651    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.785771 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.785734    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31a3434f-9205-4ec2-bb67-e0259aeb1a85-node-exporter-tls\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.797490 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.797463    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5568p\" (UniqueName: \"kubernetes.io/projected/31a3434f-9205-4ec2-bb67-e0259aeb1a85-kube-api-access-5568p\") pod \"node-exporter-kfsrc\" (UID: \"31a3434f-9205-4ec2-bb67-e0259aeb1a85\") " pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.837065 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:35.837042    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kfsrc"
May 20 15:35:35.844583 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:35:35.844563    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a3434f_9205_4ec2_bb67_e0259aeb1a85.slice/crio-7a39cf14da51605df9e3317f60d3cca3d8f83830c5cbaa995af7dfeed6e817b0 WatchSource:0}: Error finding container 7a39cf14da51605df9e3317f60d3cca3d8f83830c5cbaa995af7dfeed6e817b0: Status 404 returned error can't find the container with id 7a39cf14da51605df9e3317f60d3cca3d8f83830c5cbaa995af7dfeed6e817b0
May 20 15:35:36.388409 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:36.388369    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfsrc" event={"ID":"31a3434f-9205-4ec2-bb67-e0259aeb1a85","Type":"ContainerStarted","Data":"7a39cf14da51605df9e3317f60d3cca3d8f83830c5cbaa995af7dfeed6e817b0"}
May 20 15:35:37.393034 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:37.393002    2556 generic.go:358] "Generic (PLEG): container finished" podID="31a3434f-9205-4ec2-bb67-e0259aeb1a85" containerID="6eca3fbae44b549da23bfef6b46ab1c180214acfab3b4d04cc1163a6a5e9a0d9" exitCode=0
May 20 15:35:37.393418 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:37.393049    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfsrc" event={"ID":"31a3434f-9205-4ec2-bb67-e0259aeb1a85","Type":"ContainerDied","Data":"6eca3fbae44b549da23bfef6b46ab1c180214acfab3b4d04cc1163a6a5e9a0d9"}
May 20 15:35:38.398569 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:38.398529    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfsrc" event={"ID":"31a3434f-9205-4ec2-bb67-e0259aeb1a85","Type":"ContainerStarted","Data":"16b411703637129fb336dbc5451f0c05e80e2fb55d64a0e5a1bbdcbec6a06267"}
May 20 15:35:38.398973 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:38.398581    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfsrc" event={"ID":"31a3434f-9205-4ec2-bb67-e0259aeb1a85","Type":"ContainerStarted","Data":"ad9ce5f52f06dfce6cd5d12d7fd45f3da9641ded0086c00f617428fb3c34aaf6"}
May 20 15:35:38.450343 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:38.450286    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kfsrc" podStartSLOduration=2.511928933 podStartE2EDuration="3.45027108s" podCreationTimestamp="2026-05-20 15:35:35 +0000 UTC" firstStartedPulling="2026-05-20 15:35:35.846362315 +0000 UTC m=+178.537618221" lastFinishedPulling="2026-05-20 15:35:36.784704446 +0000 UTC m=+179.475960368" observedRunningTime="2026-05-20 15:35:38.44508967 +0000 UTC m=+181.136345597" watchObservedRunningTime="2026-05-20 15:35:38.45027108 +0000 UTC m=+181.141527020"
May 20 15:35:39.981760 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.981720    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"]
May 20 15:35:39.985012 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.984993    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:39.987978 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.987948    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-73dlrspdi48fi\""
May 20 15:35:39.989297 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.989269    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-c2m8q\""
May 20 15:35:39.989297 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.989283    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 20 15:35:39.989500 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.989303    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 20 15:35:39.989500 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.989303    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 20 15:35:39.989500 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.989277    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 20 15:35:39.996207 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:39.996182    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"]
May 20 15:35:40.118413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.118381    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6dc91e19-7fb8-4872-b89d-f5dc48522d58-metrics-server-audit-profiles\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.118522 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.118417    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-secret-metrics-server-tls\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.118522 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.118446    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc91e19-7fb8-4872-b89d-f5dc48522d58-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.118522 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.118490    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-secret-metrics-server-client-certs\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.118522 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.118519    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbfc2\" (UniqueName: \"kubernetes.io/projected/6dc91e19-7fb8-4872-b89d-f5dc48522d58-kube-api-access-qbfc2\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.118675 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.118545    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6dc91e19-7fb8-4872-b89d-f5dc48522d58-audit-log\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.118675 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.118562    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-client-ca-bundle\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.219493 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.219464    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6dc91e19-7fb8-4872-b89d-f5dc48522d58-metrics-server-audit-profiles\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.219591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.219502    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-secret-metrics-server-tls\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.219591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.219522    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc91e19-7fb8-4872-b89d-f5dc48522d58-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.219591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.219551    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-secret-metrics-server-client-certs\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.219757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.219672    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qbfc2\" (UniqueName: \"kubernetes.io/projected/6dc91e19-7fb8-4872-b89d-f5dc48522d58-kube-api-access-qbfc2\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.219757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.219710    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6dc91e19-7fb8-4872-b89d-f5dc48522d58-audit-log\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.219757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.219730    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-client-ca-bundle\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.220147 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.220124    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6dc91e19-7fb8-4872-b89d-f5dc48522d58-audit-log\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.220440 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.220418    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc91e19-7fb8-4872-b89d-f5dc48522d58-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.220624 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.220599    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6dc91e19-7fb8-4872-b89d-f5dc48522d58-metrics-server-audit-profiles\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.222277 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.222260    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-secret-metrics-server-client-certs\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.222418 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.222294    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-client-ca-bundle\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.222418 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.222389    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6dc91e19-7fb8-4872-b89d-f5dc48522d58-secret-metrics-server-tls\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.229806 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.229785    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbfc2\" (UniqueName: \"kubernetes.io/projected/6dc91e19-7fb8-4872-b89d-f5dc48522d58-kube-api-access-qbfc2\") pod \"metrics-server-6c85b47f9b-g4bkd\" (UID: \"6dc91e19-7fb8-4872-b89d-f5dc48522d58\") " pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.296139 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.296122    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:35:40.421371 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:40.421342    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"]
May 20 15:35:40.424029 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:35:40.424002    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc91e19_7fb8_4872_b89d_f5dc48522d58.slice/crio-419dea7d32555c4a2b837d97cc15bf07bde2d2a757cedaf5c2717bda3007ffd3 WatchSource:0}: Error finding container 419dea7d32555c4a2b837d97cc15bf07bde2d2a757cedaf5c2717bda3007ffd3: Status 404 returned error can't find the container with id 419dea7d32555c4a2b837d97cc15bf07bde2d2a757cedaf5c2717bda3007ffd3
May 20 15:35:41.408709 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:41.408667    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd" event={"ID":"6dc91e19-7fb8-4872-b89d-f5dc48522d58","Type":"ContainerStarted","Data":"419dea7d32555c4a2b837d97cc15bf07bde2d2a757cedaf5c2717bda3007ffd3"}
May 20 15:35:42.412537 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:42.412508    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd" event={"ID":"6dc91e19-7fb8-4872-b89d-f5dc48522d58","Type":"ContainerStarted","Data":"022b09d500ac6651f701fdc25ba8b74bead0ea441b432f0bdb784a8fee10e1bf"}
May 20 15:35:42.432455 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:42.432415    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd" podStartSLOduration=1.915109172 podStartE2EDuration="3.432404211s" podCreationTimestamp="2026-05-20 15:35:39 +0000 UTC" firstStartedPulling="2026-05-20 15:35:40.42583139 +0000 UTC m=+183.117087301" lastFinishedPulling="2026-05-20 15:35:41.94312642 +0000 UTC m=+184.634382340" observedRunningTime="2026-05-20 15:35:42.431358913 +0000 UTC m=+185.122614842" watchObservedRunningTime="2026-05-20 15:35:42.432404211 +0000 UTC m=+185.123660138"
May 20 15:35:45.355976 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:45.355946    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67948b89b7-mm82h"
May 20 15:35:48.791589 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.791554    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-88d58577d-fh82t"]
May 20 15:35:48.794728 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.794711    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:48.798396 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.798367    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 20 15:35:48.798523 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.798366    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 20 15:35:48.798523 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.798400    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 20 15:35:48.798523 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.798454    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9jwfh\""
May 20 15:35:48.798744 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.798728    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 20 15:35:48.798816 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.798753    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 20 15:35:48.798816 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.798730    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 20 15:35:48.798915 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.798769    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 20 15:35:48.804657 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.804636    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88d58577d-fh82t"]
May 20 15:35:48.976805 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.976781    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-console-config\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:48.976912 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.976818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-oauth-config\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:48.976912 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.976845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45j9\" (UniqueName: \"kubernetes.io/projected/8f2c4af3-e586-416d-80d6-da1c6564e359-kube-api-access-w45j9\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:48.976912 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.976863    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-service-ca\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:48.977030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.976948    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-oauth-serving-cert\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:48.977030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:48.976978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-serving-cert\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.077594 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.077532    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-service-ca\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.077594 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.077572    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-oauth-serving-cert\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.077594 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.077593    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-serving-cert\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.077783 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.077647    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-console-config\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.077783 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.077680    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-oauth-config\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.077783 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.077719    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w45j9\" (UniqueName: \"kubernetes.io/projected/8f2c4af3-e586-416d-80d6-da1c6564e359-kube-api-access-w45j9\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.078377 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.078346    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-service-ca\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.078377 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.078360    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-oauth-serving-cert\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.078549 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.078446    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-console-config\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.080382 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.080356    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-serving-cert\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.080382 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.080379    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-oauth-config\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.089716 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.089691    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45j9\" (UniqueName: \"kubernetes.io/projected/8f2c4af3-e586-416d-80d6-da1c6564e359-kube-api-access-w45j9\") pod \"console-88d58577d-fh82t\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") " pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.103578 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.103548    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:49.230459 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.230431    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88d58577d-fh82t"]
May 20 15:35:49.233007 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:35:49.232982    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2c4af3_e586_416d_80d6_da1c6564e359.slice/crio-f1b9f37d223880bf6fcaa85c6abb02ba41e41244ba39b6337e3aec81000115db WatchSource:0}: Error finding container f1b9f37d223880bf6fcaa85c6abb02ba41e41244ba39b6337e3aec81000115db: Status 404 returned error can't find the container with id f1b9f37d223880bf6fcaa85c6abb02ba41e41244ba39b6337e3aec81000115db
May 20 15:35:49.430873 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:49.430803    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88d58577d-fh82t" event={"ID":"8f2c4af3-e586-416d-80d6-da1c6564e359","Type":"ContainerStarted","Data":"f1b9f37d223880bf6fcaa85c6abb02ba41e41244ba39b6337e3aec81000115db"}
May 20 15:35:52.442145 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:52.442095    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88d58577d-fh82t" event={"ID":"8f2c4af3-e586-416d-80d6-da1c6564e359","Type":"ContainerStarted","Data":"ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be"}
May 20 15:35:52.463246 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:52.463195    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-88d58577d-fh82t" podStartSLOduration=1.786718886 podStartE2EDuration="4.463182151s" podCreationTimestamp="2026-05-20 15:35:48 +0000 UTC" firstStartedPulling="2026-05-20 15:35:49.234794495 +0000 UTC m=+191.926050404" lastFinishedPulling="2026-05-20 15:35:51.911257763 +0000 UTC m=+194.602513669" observedRunningTime="2026-05-20 15:35:52.46253092 +0000 UTC m=+195.153786848" watchObservedRunningTime="2026-05-20 15:35:52.463182151 +0000 UTC m=+195.154438078"
May 20 15:35:57.174265 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.174235    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cd54c6f47-54t5n"]
May 20 15:35:57.179972 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.179956    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.187699 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.187677    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 20 15:35:57.196733 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.196714    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd54c6f47-54t5n"]
May 20 15:35:57.238159 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.238138    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-console-config\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.238250 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.238168    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-oauth-serving-cert\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.238250 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.238191    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-service-ca\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.238332 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.238243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-serving-cert\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.238332 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.238285    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-trusted-ca-bundle\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.238406 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.238341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-oauth-config\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.238406 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.238371    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drfgc\" (UniqueName: \"kubernetes.io/projected/b993694b-e004-42d2-8f71-6c81a29d36ff-kube-api-access-drfgc\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.338628 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.338598    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-service-ca\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.338722 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.338632    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-serving-cert\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.338722 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.338657    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-trusted-ca-bundle\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.338722 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.338700    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-oauth-config\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.338887 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.338740    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-drfgc\" (UniqueName: \"kubernetes.io/projected/b993694b-e004-42d2-8f71-6c81a29d36ff-kube-api-access-drfgc\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.338945 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.338907    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-console-config\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.339001 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.338968    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-oauth-serving-cert\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.339413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.339389    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-service-ca\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.339602 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.339582    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-trusted-ca-bundle\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.339678 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.339627    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-console-config\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.339740 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.339692    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-oauth-serving-cert\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.341353 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.341332    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-serving-cert\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.341746 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.341729    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-oauth-config\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.347466 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.347434    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drfgc\" (UniqueName: \"kubernetes.io/projected/b993694b-e004-42d2-8f71-6c81a29d36ff-kube-api-access-drfgc\") pod \"console-6cd54c6f47-54t5n\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") " pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.488505 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.488481    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:35:57.615594 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:57.615566    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd54c6f47-54t5n"]
May 20 15:35:57.619022 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:35:57.618990    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb993694b_e004_42d2_8f71_6c81a29d36ff.slice/crio-03c6d46b88a61e12602b9aff112d40737e3d08c496095edeff9accffd7bf982c WatchSource:0}: Error finding container 03c6d46b88a61e12602b9aff112d40737e3d08c496095edeff9accffd7bf982c: Status 404 returned error can't find the container with id 03c6d46b88a61e12602b9aff112d40737e3d08c496095edeff9accffd7bf982c
May 20 15:35:58.457701 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:58.457661    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd54c6f47-54t5n" event={"ID":"b993694b-e004-42d2-8f71-6c81a29d36ff","Type":"ContainerStarted","Data":"1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc"}
May 20 15:35:58.457701 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:58.457699    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd54c6f47-54t5n" event={"ID":"b993694b-e004-42d2-8f71-6c81a29d36ff","Type":"ContainerStarted","Data":"03c6d46b88a61e12602b9aff112d40737e3d08c496095edeff9accffd7bf982c"}
May 20 15:35:58.480738 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:58.480693    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6cd54c6f47-54t5n" podStartSLOduration=1.4806804439999999 podStartE2EDuration="1.480680444s" podCreationTimestamp="2026-05-20 15:35:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:35:58.479907894 +0000 UTC m=+201.171163822" watchObservedRunningTime="2026-05-20 15:35:58.480680444 +0000 UTC m=+201.171936372"
May 20 15:35:59.104353 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:59.104301    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:59.104353 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:59.104361    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:59.109335 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:59.109310    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88d58577d-fh82t"
May 20 15:35:59.464310 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:35:59.464245    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88d58577d-fh82t"
May 20 15:36:00.296842 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:00.296805    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:36:00.297030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:00.296890    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:36:03.295008 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:03.294961    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" podUID="4c113c03-3b96-4e91-843d-7c7ce941e8bd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 20 15:36:07.489253 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:07.489218    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:36:07.489253 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:07.489258    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:36:07.493534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:07.493512    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:36:08.486390 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:08.486363    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:36:08.553075 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:08.553049    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88d58577d-fh82t"]
May 20 15:36:13.295308 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:13.295271    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" podUID="4c113c03-3b96-4e91-843d-7c7ce941e8bd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 20 15:36:20.302864 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:20.302823    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:36:20.306879 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:20.306850    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c85b47f9b-g4bkd"
May 20 15:36:23.294634 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:23.294602    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" podUID="4c113c03-3b96-4e91-843d-7c7ce941e8bd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 20 15:36:23.295081 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:23.294684    2556 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl"
May 20 15:36:23.295198 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:23.295133    2556 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"7be0b02bcb5b99dbfa955baeca62b7bc9806365fd219812c8e34eb020930df95"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 20 15:36:23.295198 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:23.295179    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" podUID="4c113c03-3b96-4e91-843d-7c7ce941e8bd" containerName="service-proxy" containerID="cri-o://7be0b02bcb5b99dbfa955baeca62b7bc9806365fd219812c8e34eb020930df95" gracePeriod=30
May 20 15:36:23.523423 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:23.523390    2556 generic.go:358] "Generic (PLEG): container finished" podID="4c113c03-3b96-4e91-843d-7c7ce941e8bd" containerID="7be0b02bcb5b99dbfa955baeca62b7bc9806365fd219812c8e34eb020930df95" exitCode=2
May 20 15:36:23.523566 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:23.523458    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" event={"ID":"4c113c03-3b96-4e91-843d-7c7ce941e8bd","Type":"ContainerDied","Data":"7be0b02bcb5b99dbfa955baeca62b7bc9806365fd219812c8e34eb020930df95"}
May 20 15:36:23.523566 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:23.523493    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-784d9ff8ff-68vzl" event={"ID":"4c113c03-3b96-4e91-843d-7c7ce941e8bd","Type":"ContainerStarted","Data":"1f5b17bd117464d7ebb1639610c51481e15333309f1a438ab2fd44a663fd4f47"}
May 20 15:36:33.572220 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.572168    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-88d58577d-fh82t" podUID="8f2c4af3-e586-416d-80d6-da1c6564e359" containerName="console" containerID="cri-o://ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be" gracePeriod=15
May 20 15:36:33.801265 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.801246    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-88d58577d-fh82t_8f2c4af3-e586-416d-80d6-da1c6564e359/console/0.log"
May 20 15:36:33.801365 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.801312    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88d58577d-fh82t"
May 20 15:36:33.901254 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901177    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w45j9\" (UniqueName: \"kubernetes.io/projected/8f2c4af3-e586-416d-80d6-da1c6564e359-kube-api-access-w45j9\") pod \"8f2c4af3-e586-416d-80d6-da1c6564e359\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") "
May 20 15:36:33.901254 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901209    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-service-ca\") pod \"8f2c4af3-e586-416d-80d6-da1c6564e359\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") "
May 20 15:36:33.901254 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901226    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-oauth-serving-cert\") pod \"8f2c4af3-e586-416d-80d6-da1c6564e359\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") "
May 20 15:36:33.901524 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901277    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-serving-cert\") pod \"8f2c4af3-e586-416d-80d6-da1c6564e359\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") "
May 20 15:36:33.901524 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901308    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-oauth-config\") pod \"8f2c4af3-e586-416d-80d6-da1c6564e359\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") "
May 20 15:36:33.901524 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901344    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-console-config\") pod \"8f2c4af3-e586-416d-80d6-da1c6564e359\" (UID: \"8f2c4af3-e586-416d-80d6-da1c6564e359\") "
May 20 15:36:33.901745 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901717    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f2c4af3-e586-416d-80d6-da1c6564e359" (UID: "8f2c4af3-e586-416d-80d6-da1c6564e359"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:36:33.901865 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901762    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f2c4af3-e586-416d-80d6-da1c6564e359" (UID: "8f2c4af3-e586-416d-80d6-da1c6564e359"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:36:33.901865 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.901825    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-console-config" (OuterVolumeSpecName: "console-config") pod "8f2c4af3-e586-416d-80d6-da1c6564e359" (UID: "8f2c4af3-e586-416d-80d6-da1c6564e359"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:36:33.903479 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.903458    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f2c4af3-e586-416d-80d6-da1c6564e359" (UID: "8f2c4af3-e586-416d-80d6-da1c6564e359"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:36:33.903562 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.903532    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2c4af3-e586-416d-80d6-da1c6564e359-kube-api-access-w45j9" (OuterVolumeSpecName: "kube-api-access-w45j9") pod "8f2c4af3-e586-416d-80d6-da1c6564e359" (UID: "8f2c4af3-e586-416d-80d6-da1c6564e359"). InnerVolumeSpecName "kube-api-access-w45j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:36:33.903603 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:33.903589    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f2c4af3-e586-416d-80d6-da1c6564e359" (UID: "8f2c4af3-e586-416d-80d6-da1c6564e359"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:36:34.002782 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.002751    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45j9\" (UniqueName: \"kubernetes.io/projected/8f2c4af3-e586-416d-80d6-da1c6564e359-kube-api-access-w45j9\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:36:34.002782 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.002775    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-service-ca\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:36:34.002782 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.002786    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-oauth-serving-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:36:34.002988 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.002796    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-serving-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:36:34.002988 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.002805    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f2c4af3-e586-416d-80d6-da1c6564e359-console-oauth-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:36:34.002988 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.002814    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f2c4af3-e586-416d-80d6-da1c6564e359-console-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:36:34.551668 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.551644    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-88d58577d-fh82t_8f2c4af3-e586-416d-80d6-da1c6564e359/console/0.log"
May 20 15:36:34.551825 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.551684    2556 generic.go:358] "Generic (PLEG): container finished" podID="8f2c4af3-e586-416d-80d6-da1c6564e359" containerID="ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be" exitCode=2
May 20 15:36:34.551825 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.551755    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88d58577d-fh82t"
May 20 15:36:34.551825 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.551768    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88d58577d-fh82t" event={"ID":"8f2c4af3-e586-416d-80d6-da1c6564e359","Type":"ContainerDied","Data":"ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be"}
May 20 15:36:34.551825 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.551803    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88d58577d-fh82t" event={"ID":"8f2c4af3-e586-416d-80d6-da1c6564e359","Type":"ContainerDied","Data":"f1b9f37d223880bf6fcaa85c6abb02ba41e41244ba39b6337e3aec81000115db"}
May 20 15:36:34.551825 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.551826    2556 scope.go:117] "RemoveContainer" containerID="ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be"
May 20 15:36:34.560155 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.560140    2556 scope.go:117] "RemoveContainer" containerID="ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be"
May 20 15:36:34.560396 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:36:34.560378    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be\": container with ID starting with ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be not found: ID does not exist" containerID="ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be"
May 20 15:36:34.560454 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.560403    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be"} err="failed to get container status \"ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be\": rpc error: code = NotFound desc = could not find container \"ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be\": container with ID starting with ce610a3788ff36f1f9061982e2fbaeaefda4266d3c1581e1b9d6ebd5c02085be not found: ID does not exist"
May 20 15:36:34.576534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.576515    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88d58577d-fh82t"]
May 20 15:36:34.581616 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:34.581592    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88d58577d-fh82t"]
May 20 15:36:35.890564 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:35.890531    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2c4af3-e586-416d-80d6-da1c6564e359" path="/var/lib/kubelet/pods/8f2c4af3-e586-416d-80d6-da1c6564e359/volumes"
May 20 15:36:49.620680 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:49.620654    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:36:49.622851 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:49.622825    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67e163d-ec7d-4d41-b2b6-27e8bbab3825-metrics-certs\") pod \"network-metrics-daemon-44pnc\" (UID: \"e67e163d-ec7d-4d41-b2b6-27e8bbab3825\") " pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:36:49.693374 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:49.693355    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-79qfs\""
May 20 15:36:49.701544 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:49.701528    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44pnc"
May 20 15:36:49.822095 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:49.822071    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-44pnc"]
May 20 15:36:49.824708 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:36:49.824680    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67e163d_ec7d_4d41_b2b6_27e8bbab3825.slice/crio-b8b71d43ca5f947d95450821beaef162011065a445b3fed89c2649f5d2b1292f WatchSource:0}: Error finding container b8b71d43ca5f947d95450821beaef162011065a445b3fed89c2649f5d2b1292f: Status 404 returned error can't find the container with id b8b71d43ca5f947d95450821beaef162011065a445b3fed89c2649f5d2b1292f
May 20 15:36:50.597118 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:50.597059    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-44pnc" event={"ID":"e67e163d-ec7d-4d41-b2b6-27e8bbab3825","Type":"ContainerStarted","Data":"b8b71d43ca5f947d95450821beaef162011065a445b3fed89c2649f5d2b1292f"}
May 20 15:36:51.602438 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:51.602396    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-44pnc" event={"ID":"e67e163d-ec7d-4d41-b2b6-27e8bbab3825","Type":"ContainerStarted","Data":"1367d59a847f4f7bbed7323578e238d0532f38c858c65362f66b7d7f19c85da6"}
May 20 15:36:51.602438 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:51.602439    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-44pnc" event={"ID":"e67e163d-ec7d-4d41-b2b6-27e8bbab3825","Type":"ContainerStarted","Data":"b051adfb668e92e2e9f3a0b3bd850a131cb8ecc2dc5ab62710e0da4d3691db84"}
May 20 15:36:51.631967 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:51.631916    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-44pnc" podStartSLOduration=253.682851811 podStartE2EDuration="4m14.631900709s" podCreationTimestamp="2026-05-20 15:32:37 +0000 UTC" firstStartedPulling="2026-05-20 15:36:49.826850104 +0000 UTC m=+252.518106010" lastFinishedPulling="2026-05-20 15:36:50.775898998 +0000 UTC m=+253.467154908" observedRunningTime="2026-05-20 15:36:51.630512029 +0000 UTC m=+254.321767958" watchObservedRunningTime="2026-05-20 15:36:51.631900709 +0000 UTC m=+254.323156637"
May 20 15:36:55.520726 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.520695    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5547964f64-5fhz9"]
May 20 15:36:55.521202 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.520930    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2c4af3-e586-416d-80d6-da1c6564e359" containerName="console"
May 20 15:36:55.521202 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.520943    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2c4af3-e586-416d-80d6-da1c6564e359" containerName="console"
May 20 15:36:55.521202 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.520990    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f2c4af3-e586-416d-80d6-da1c6564e359" containerName="console"
May 20 15:36:55.522897 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.522876    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.554090 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.554062    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5547964f64-5fhz9"]
May 20 15:36:55.669608 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.669585    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-config\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.669711 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.669618    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-service-ca\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.669711 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.669648    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-serving-cert\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.669808 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.669711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-oauth-serving-cert\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.669808 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.669765    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-trusted-ca-bundle\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.669808 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.669792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psggd\" (UniqueName: \"kubernetes.io/projected/831dbc8c-bc07-413c-b068-ea6eb8dd725d-kube-api-access-psggd\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.669921 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.669819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-oauth-config\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771023 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.770953    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-oauth-config\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771023 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771011    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-config\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771210 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771043    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-service-ca\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771210 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771076    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-serving-cert\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771210 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771140    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-oauth-serving-cert\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771210 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771183    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-trusted-ca-bundle\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771404 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771213    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-psggd\" (UniqueName: \"kubernetes.io/projected/831dbc8c-bc07-413c-b068-ea6eb8dd725d-kube-api-access-psggd\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771799 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771779    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-config\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771906 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771799    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-service-ca\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.771906 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.771884    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-oauth-serving-cert\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.772199 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.772181    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-trusted-ca-bundle\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.773437 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.773415    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-oauth-config\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.773537 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.773513    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-serving-cert\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.779785 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.779767    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psggd\" (UniqueName: \"kubernetes.io/projected/831dbc8c-bc07-413c-b068-ea6eb8dd725d-kube-api-access-psggd\") pod \"console-5547964f64-5fhz9\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") " pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.831806 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.831781    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:36:55.963961 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:55.963935    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5547964f64-5fhz9"]
May 20 15:36:55.966332 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:36:55.966306    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831dbc8c_bc07_413c_b068_ea6eb8dd725d.slice/crio-9a6fa3d5421ea9780f405fd8a2f3d199c3292e6cee737ec776e2ae457e6c7b39 WatchSource:0}: Error finding container 9a6fa3d5421ea9780f405fd8a2f3d199c3292e6cee737ec776e2ae457e6c7b39: Status 404 returned error can't find the container with id 9a6fa3d5421ea9780f405fd8a2f3d199c3292e6cee737ec776e2ae457e6c7b39
May 20 15:36:56.617897 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:56.617860    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5547964f64-5fhz9" event={"ID":"831dbc8c-bc07-413c-b068-ea6eb8dd725d","Type":"ContainerStarted","Data":"4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b"}
May 20 15:36:56.617897 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:56.617900    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5547964f64-5fhz9" event={"ID":"831dbc8c-bc07-413c-b068-ea6eb8dd725d","Type":"ContainerStarted","Data":"9a6fa3d5421ea9780f405fd8a2f3d199c3292e6cee737ec776e2ae457e6c7b39"}
May 20 15:36:56.639287 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:36:56.639244    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5547964f64-5fhz9" podStartSLOduration=1.639230548 podStartE2EDuration="1.639230548s" podCreationTimestamp="2026-05-20 15:36:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:36:56.638754548 +0000 UTC m=+259.330010477" watchObservedRunningTime="2026-05-20 15:36:56.639230548 +0000 UTC m=+259.330486476"
May 20 15:37:05.832637 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:05.832592    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:37:05.833158 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:05.832651    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:37:05.837652 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:05.837624    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:37:06.649773 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:06.649740    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:37:06.707053 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:06.707027    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd54c6f47-54t5n"]
May 20 15:37:31.725971 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:31.725911    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6cd54c6f47-54t5n" podUID="b993694b-e004-42d2-8f71-6c81a29d36ff" containerName="console" containerID="cri-o://1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc" gracePeriod=15
May 20 15:37:31.961721 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:31.961698    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cd54c6f47-54t5n_b993694b-e004-42d2-8f71-6c81a29d36ff/console/0.log"
May 20 15:37:31.961830 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:31.961755    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:37:32.015816 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.015787    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-console-config\") pod \"b993694b-e004-42d2-8f71-6c81a29d36ff\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") "
May 20 15:37:32.015816 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.015816    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-trusted-ca-bundle\") pod \"b993694b-e004-42d2-8f71-6c81a29d36ff\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") "
May 20 15:37:32.016029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.015838    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-service-ca\") pod \"b993694b-e004-42d2-8f71-6c81a29d36ff\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") "
May 20 15:37:32.016029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.015867    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-serving-cert\") pod \"b993694b-e004-42d2-8f71-6c81a29d36ff\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") "
May 20 15:37:32.016029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.015886    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-oauth-config\") pod \"b993694b-e004-42d2-8f71-6c81a29d36ff\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") "
May 20 15:37:32.016029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.015926    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-oauth-serving-cert\") pod \"b993694b-e004-42d2-8f71-6c81a29d36ff\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") "
May 20 15:37:32.016029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.015963    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drfgc\" (UniqueName: \"kubernetes.io/projected/b993694b-e004-42d2-8f71-6c81a29d36ff-kube-api-access-drfgc\") pod \"b993694b-e004-42d2-8f71-6c81a29d36ff\" (UID: \"b993694b-e004-42d2-8f71-6c81a29d36ff\") "
May 20 15:37:32.016299 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.016195    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-console-config" (OuterVolumeSpecName: "console-config") pod "b993694b-e004-42d2-8f71-6c81a29d36ff" (UID: "b993694b-e004-42d2-8f71-6c81a29d36ff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:37:32.016299 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.016244    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-service-ca" (OuterVolumeSpecName: "service-ca") pod "b993694b-e004-42d2-8f71-6c81a29d36ff" (UID: "b993694b-e004-42d2-8f71-6c81a29d36ff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:37:32.016382 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.016290    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b993694b-e004-42d2-8f71-6c81a29d36ff" (UID: "b993694b-e004-42d2-8f71-6c81a29d36ff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:37:32.016382 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.016318    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b993694b-e004-42d2-8f71-6c81a29d36ff" (UID: "b993694b-e004-42d2-8f71-6c81a29d36ff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:37:32.017974 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.017940    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b993694b-e004-42d2-8f71-6c81a29d36ff" (UID: "b993694b-e004-42d2-8f71-6c81a29d36ff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:37:32.018153 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.018126    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b993694b-e004-42d2-8f71-6c81a29d36ff-kube-api-access-drfgc" (OuterVolumeSpecName: "kube-api-access-drfgc") pod "b993694b-e004-42d2-8f71-6c81a29d36ff" (UID: "b993694b-e004-42d2-8f71-6c81a29d36ff"). InnerVolumeSpecName "kube-api-access-drfgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:37:32.018266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.018171    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b993694b-e004-42d2-8f71-6c81a29d36ff" (UID: "b993694b-e004-42d2-8f71-6c81a29d36ff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:37:32.116568 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.116546    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-console-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:37:32.116568 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.116564    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-trusted-ca-bundle\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:37:32.116686 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.116575    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-service-ca\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:37:32.116686 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.116584    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-serving-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:37:32.116686 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.116592    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b993694b-e004-42d2-8f71-6c81a29d36ff-console-oauth-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:37:32.116686 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.116600    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b993694b-e004-42d2-8f71-6c81a29d36ff-oauth-serving-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:37:32.116686 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.116608    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drfgc\" (UniqueName: \"kubernetes.io/projected/b993694b-e004-42d2-8f71-6c81a29d36ff-kube-api-access-drfgc\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:37:32.715235 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.715207    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cd54c6f47-54t5n_b993694b-e004-42d2-8f71-6c81a29d36ff/console/0.log"
May 20 15:37:32.715437 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.715249    2556 generic.go:358] "Generic (PLEG): container finished" podID="b993694b-e004-42d2-8f71-6c81a29d36ff" containerID="1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc" exitCode=2
May 20 15:37:32.715437 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.715287    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd54c6f47-54t5n" event={"ID":"b993694b-e004-42d2-8f71-6c81a29d36ff","Type":"ContainerDied","Data":"1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc"}
May 20 15:37:32.715437 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.715322    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd54c6f47-54t5n"
May 20 15:37:32.715437 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.715338    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd54c6f47-54t5n" event={"ID":"b993694b-e004-42d2-8f71-6c81a29d36ff","Type":"ContainerDied","Data":"03c6d46b88a61e12602b9aff112d40737e3d08c496095edeff9accffd7bf982c"}
May 20 15:37:32.715437 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.715359    2556 scope.go:117] "RemoveContainer" containerID="1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc"
May 20 15:37:32.724185 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.724168    2556 scope.go:117] "RemoveContainer" containerID="1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc"
May 20 15:37:32.724454 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:37:32.724434    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc\": container with ID starting with 1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc not found: ID does not exist" containerID="1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc"
May 20 15:37:32.724563 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.724464    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc"} err="failed to get container status \"1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc\": rpc error: code = NotFound desc = could not find container \"1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc\": container with ID starting with 1428460a7a89247ca28c2aba02d5a5cb5154dce0cd78e4b07f64f9031396c8dc not found: ID does not exist"
May 20 15:37:32.740171 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.740144    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd54c6f47-54t5n"]
May 20 15:37:32.747242 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:32.747221    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cd54c6f47-54t5n"]
May 20 15:37:33.890153 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:33.890094    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b993694b-e004-42d2-8f71-6c81a29d36ff" path="/var/lib/kubelet/pods/b993694b-e004-42d2-8f71-6c81a29d36ff/volumes"
May 20 15:37:37.746304 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:37:37.746017    2556 kubelet.go:1628] "Image garbage collection succeeded"
May 20 15:39:49.349822 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.349789    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"]
May 20 15:39:49.350303 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.350021    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b993694b-e004-42d2-8f71-6c81a29d36ff" containerName="console"
May 20 15:39:49.350303 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.350031    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b993694b-e004-42d2-8f71-6c81a29d36ff" containerName="console"
May 20 15:39:49.350303 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.350095    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="b993694b-e004-42d2-8f71-6c81a29d36ff" containerName="console"
May 20 15:39:49.352774 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.352751    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:49.355578 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.355554    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 15:39:49.355707 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.355626    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-b6ffz\""
May 20 15:39:49.356652 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.356624    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 20 15:39:49.356747 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.356667    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 15:39:49.368314 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.368291    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"]
May 20 15:39:49.452780 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.452737    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01a55bae-a072-4d7d-ada8-9f1670136ef7-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-xwx29\" (UID: \"01a55bae-a072-4d7d-ada8-9f1670136ef7\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:49.452947 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.452845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmdm\" (UniqueName: \"kubernetes.io/projected/01a55bae-a072-4d7d-ada8-9f1670136ef7-kube-api-access-8kmdm\") pod \"llmisvc-controller-manager-68cc5db7c4-xwx29\" (UID: \"01a55bae-a072-4d7d-ada8-9f1670136ef7\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:49.553434 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.553392    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmdm\" (UniqueName: \"kubernetes.io/projected/01a55bae-a072-4d7d-ada8-9f1670136ef7-kube-api-access-8kmdm\") pod \"llmisvc-controller-manager-68cc5db7c4-xwx29\" (UID: \"01a55bae-a072-4d7d-ada8-9f1670136ef7\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:49.553605 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.553456    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01a55bae-a072-4d7d-ada8-9f1670136ef7-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-xwx29\" (UID: \"01a55bae-a072-4d7d-ada8-9f1670136ef7\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:49.555957 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.555934    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01a55bae-a072-4d7d-ada8-9f1670136ef7-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-xwx29\" (UID: \"01a55bae-a072-4d7d-ada8-9f1670136ef7\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:49.564871 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.564845    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmdm\" (UniqueName: \"kubernetes.io/projected/01a55bae-a072-4d7d-ada8-9f1670136ef7-kube-api-access-8kmdm\") pod \"llmisvc-controller-manager-68cc5db7c4-xwx29\" (UID: \"01a55bae-a072-4d7d-ada8-9f1670136ef7\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:49.663199 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.663093    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:49.785229 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.785198    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"]
May 20 15:39:49.788749 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:39:49.788720    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod01a55bae_a072_4d7d_ada8_9f1670136ef7.slice/crio-7ce7e31df5b0c96342ea15eb8d729b87d09bdab7f5d259d16d0616776f28bc36 WatchSource:0}: Error finding container 7ce7e31df5b0c96342ea15eb8d729b87d09bdab7f5d259d16d0616776f28bc36: Status 404 returned error can't find the container with id 7ce7e31df5b0c96342ea15eb8d729b87d09bdab7f5d259d16d0616776f28bc36
May 20 15:39:49.789888 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:49.789872    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:39:50.064890 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:50.064850    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29" event={"ID":"01a55bae-a072-4d7d-ada8-9f1670136ef7","Type":"ContainerStarted","Data":"7ce7e31df5b0c96342ea15eb8d729b87d09bdab7f5d259d16d0616776f28bc36"}
May 20 15:39:53.074153 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:53.074114    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29" event={"ID":"01a55bae-a072-4d7d-ada8-9f1670136ef7","Type":"ContainerStarted","Data":"a6b7ded88d253ef666c91b7afbb38531fe46244cd792684fd69bd6a6d3dd6646"}
May 20 15:39:53.074564 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:53.074262    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:39:53.092900 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:39:53.092854    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29" podStartSLOduration=1.836571344 podStartE2EDuration="4.092841125s" podCreationTimestamp="2026-05-20 15:39:49 +0000 UTC" firstStartedPulling="2026-05-20 15:39:49.789985527 +0000 UTC m=+432.481241434" lastFinishedPulling="2026-05-20 15:39:52.046255308 +0000 UTC m=+434.737511215" observedRunningTime="2026-05-20 15:39:53.091466388 +0000 UTC m=+435.782722317" watchObservedRunningTime="2026-05-20 15:39:53.092841125 +0000 UTC m=+435.784097067"
May 20 15:40:24.078734 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:40:24.078703    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-68cc5db7c4-xwx29"
May 20 15:41:05.550476 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.550441    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-bdc794bc6-gdfr9"]
May 20 15:41:05.553337 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.553316    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.571634 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.571608    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdc794bc6-gdfr9"]
May 20 15:41:05.714561 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.714517    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-oauth-config\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.714738 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.714581    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-oauth-serving-cert\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.714738 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.714653    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-service-ca\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.714738 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.714690    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-config\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.714738 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.714716    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-serving-cert\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.714872 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.714740    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-trusted-ca-bundle\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.714872 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.714764    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84slj\" (UniqueName: \"kubernetes.io/projected/90f39d67-2aee-42c2-b935-2bfd1eae06bc-kube-api-access-84slj\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.815493 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.815405    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-service-ca\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.815493 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.815448    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-config\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.815493 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.815468    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-serving-cert\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.815493 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.815484    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-trusted-ca-bundle\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.815740 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.815516    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-84slj\" (UniqueName: \"kubernetes.io/projected/90f39d67-2aee-42c2-b935-2bfd1eae06bc-kube-api-access-84slj\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.815740 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.815559    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-oauth-config\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.815740 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.815636    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-oauth-serving-cert\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.816283 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.816258    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-service-ca\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.816386 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.816298    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-config\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.816433 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.816391    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-oauth-serving-cert\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.816507 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.816432    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90f39d67-2aee-42c2-b935-2bfd1eae06bc-trusted-ca-bundle\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.817990 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.817971    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-serving-cert\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.818134 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.818112    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90f39d67-2aee-42c2-b935-2bfd1eae06bc-console-oauth-config\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.824761 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.824741    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84slj\" (UniqueName: \"kubernetes.io/projected/90f39d67-2aee-42c2-b935-2bfd1eae06bc-kube-api-access-84slj\") pod \"console-bdc794bc6-gdfr9\" (UID: \"90f39d67-2aee-42c2-b935-2bfd1eae06bc\") " pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.865395 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.865365    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:05.987006 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:05.986831    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdc794bc6-gdfr9"]
May 20 15:41:05.989548 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:41:05.989525    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f39d67_2aee_42c2_b935_2bfd1eae06bc.slice/crio-93359ce0431f9f8a8aee201af5f7141703c7f5f5a2c39b48bf0b94476f6aabca WatchSource:0}: Error finding container 93359ce0431f9f8a8aee201af5f7141703c7f5f5a2c39b48bf0b94476f6aabca: Status 404 returned error can't find the container with id 93359ce0431f9f8a8aee201af5f7141703c7f5f5a2c39b48bf0b94476f6aabca
May 20 15:41:06.259344 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:06.259309    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc794bc6-gdfr9" event={"ID":"90f39d67-2aee-42c2-b935-2bfd1eae06bc","Type":"ContainerStarted","Data":"89d9ba006dbf5fa083f53fd065957473c2db698099b5be53194e768d935fd9dc"}
May 20 15:41:06.259344 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:06.259347    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc794bc6-gdfr9" event={"ID":"90f39d67-2aee-42c2-b935-2bfd1eae06bc","Type":"ContainerStarted","Data":"93359ce0431f9f8a8aee201af5f7141703c7f5f5a2c39b48bf0b94476f6aabca"}
May 20 15:41:06.283328 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:06.283284    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-bdc794bc6-gdfr9" podStartSLOduration=1.283271403 podStartE2EDuration="1.283271403s" podCreationTimestamp="2026-05-20 15:41:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:41:06.281318287 +0000 UTC m=+508.972574213" watchObservedRunningTime="2026-05-20 15:41:06.283271403 +0000 UTC m=+508.974527330"
May 20 15:41:15.865713 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:15.865679    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:15.865713 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:15.865718    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:15.870360 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:15.870340    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:16.288340 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:16.288312    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bdc794bc6-gdfr9"
May 20 15:41:16.346246 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:16.346216    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5547964f64-5fhz9"]
May 20 15:41:41.364587 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.364494    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5547964f64-5fhz9" podUID="831dbc8c-bc07-413c-b068-ea6eb8dd725d" containerName="console" containerID="cri-o://4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b" gracePeriod=15
May 20 15:41:41.595981 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.595954    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5547964f64-5fhz9_831dbc8c-bc07-413c-b068-ea6eb8dd725d/console/0.log"
May 20 15:41:41.596083 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.596013    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:41:41.671171 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671111    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-serving-cert\") pod \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") "
May 20 15:41:41.671171 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671143    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-oauth-config\") pod \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") "
May 20 15:41:41.671171 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671165    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-trusted-ca-bundle\") pod \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") "
May 20 15:41:41.671357 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671193    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psggd\" (UniqueName: \"kubernetes.io/projected/831dbc8c-bc07-413c-b068-ea6eb8dd725d-kube-api-access-psggd\") pod \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") "
May 20 15:41:41.671357 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671299    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-config\") pod \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") "
May 20 15:41:41.671460 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671360    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-oauth-serving-cert\") pod \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") "
May 20 15:41:41.671460 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671397    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-service-ca\") pod \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\" (UID: \"831dbc8c-bc07-413c-b068-ea6eb8dd725d\") "
May 20 15:41:41.671684 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671657    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "831dbc8c-bc07-413c-b068-ea6eb8dd725d" (UID: "831dbc8c-bc07-413c-b068-ea6eb8dd725d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:41.671684 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671661    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-config" (OuterVolumeSpecName: "console-config") pod "831dbc8c-bc07-413c-b068-ea6eb8dd725d" (UID: "831dbc8c-bc07-413c-b068-ea6eb8dd725d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:41.671853 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671813    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "831dbc8c-bc07-413c-b068-ea6eb8dd725d" (UID: "831dbc8c-bc07-413c-b068-ea6eb8dd725d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:41.671898 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.671869    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-service-ca" (OuterVolumeSpecName: "service-ca") pod "831dbc8c-bc07-413c-b068-ea6eb8dd725d" (UID: "831dbc8c-bc07-413c-b068-ea6eb8dd725d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:41:41.673320 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.673293    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "831dbc8c-bc07-413c-b068-ea6eb8dd725d" (UID: "831dbc8c-bc07-413c-b068-ea6eb8dd725d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:41:41.673320 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.673314    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831dbc8c-bc07-413c-b068-ea6eb8dd725d-kube-api-access-psggd" (OuterVolumeSpecName: "kube-api-access-psggd") pod "831dbc8c-bc07-413c-b068-ea6eb8dd725d" (UID: "831dbc8c-bc07-413c-b068-ea6eb8dd725d"). InnerVolumeSpecName "kube-api-access-psggd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:41:41.673457 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.673368    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "831dbc8c-bc07-413c-b068-ea6eb8dd725d" (UID: "831dbc8c-bc07-413c-b068-ea6eb8dd725d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:41:41.772617 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.772593    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-oauth-serving-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:41:41.772617 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.772615    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-service-ca\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:41:41.772754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.772630    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-serving-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:41:41.772754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.772643    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-oauth-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:41:41.772754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.772656    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-trusted-ca-bundle\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:41:41.772754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.772669    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psggd\" (UniqueName: \"kubernetes.io/projected/831dbc8c-bc07-413c-b068-ea6eb8dd725d-kube-api-access-psggd\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:41:41.772754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:41.772683    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/831dbc8c-bc07-413c-b068-ea6eb8dd725d-console-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:41:42.350385 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.350361    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5547964f64-5fhz9_831dbc8c-bc07-413c-b068-ea6eb8dd725d/console/0.log"
May 20 15:41:42.350542 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.350401    2556 generic.go:358] "Generic (PLEG): container finished" podID="831dbc8c-bc07-413c-b068-ea6eb8dd725d" containerID="4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b" exitCode=2
May 20 15:41:42.350542 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.350469    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5547964f64-5fhz9"
May 20 15:41:42.350542 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.350477    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5547964f64-5fhz9" event={"ID":"831dbc8c-bc07-413c-b068-ea6eb8dd725d","Type":"ContainerDied","Data":"4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b"}
May 20 15:41:42.350542 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.350504    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5547964f64-5fhz9" event={"ID":"831dbc8c-bc07-413c-b068-ea6eb8dd725d","Type":"ContainerDied","Data":"9a6fa3d5421ea9780f405fd8a2f3d199c3292e6cee737ec776e2ae457e6c7b39"}
May 20 15:41:42.350542 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.350522    2556 scope.go:117] "RemoveContainer" containerID="4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b"
May 20 15:41:42.358647 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.358630    2556 scope.go:117] "RemoveContainer" containerID="4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b"
May 20 15:41:42.358903 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:41:42.358879    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b\": container with ID starting with 4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b not found: ID does not exist" containerID="4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b"
May 20 15:41:42.358993 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.358909    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b"} err="failed to get container status \"4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b\": rpc error: code = NotFound desc = could not find container \"4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b\": container with ID starting with 4bab2fcae3f41b2d66342736ed2880afa08421d16d93cfa652d9fa680a0e665b not found: ID does not exist"
May 20 15:41:42.371443 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.371422    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5547964f64-5fhz9"]
May 20 15:41:42.375928 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:42.375908    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5547964f64-5fhz9"]
May 20 15:41:43.890740 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:41:43.890708    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="831dbc8c-bc07-413c-b068-ea6eb8dd725d" path="/var/lib/kubelet/pods/831dbc8c-bc07-413c-b068-ea6eb8dd725d/volumes"
May 20 15:42:32.670381 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.670350    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"]
May 20 15:42:32.670794 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.670611    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="831dbc8c-bc07-413c-b068-ea6eb8dd725d" containerName="console"
May 20 15:42:32.670794 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.670622    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="831dbc8c-bc07-413c-b068-ea6eb8dd725d" containerName="console"
May 20 15:42:32.670794 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.670673    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="831dbc8c-bc07-413c-b068-ea6eb8dd725d" containerName="console"
May 20 15:42:32.673810 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.673793    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.678268 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.678241    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
May 20 15:42:32.678395 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.678277    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:42:32.678395 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.678315    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fxjhf\""
May 20 15:42:32.678395 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.678315    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:42:32.678728 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.678713    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
May 20 15:42:32.689329 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.689309    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"]
May 20 15:42:32.828037 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.828014    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.828177 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.828051    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918bdba5-698f-4a29-8ff8-ee0012f9e70a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.828177 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.828082    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918bdba5-698f-4a29-8ff8-ee0012f9e70a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.828177 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.828142    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w777t\" (UniqueName: \"kubernetes.io/projected/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kube-api-access-w777t\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.928631 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.928574    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.928631 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.928607    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918bdba5-698f-4a29-8ff8-ee0012f9e70a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.928783 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.928634    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918bdba5-698f-4a29-8ff8-ee0012f9e70a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.928783 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.928657    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w777t\" (UniqueName: \"kubernetes.io/projected/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kube-api-access-w777t\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.929037 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.929016    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.929364 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.929344    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918bdba5-698f-4a29-8ff8-ee0012f9e70a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.930970 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.930955    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918bdba5-698f-4a29-8ff8-ee0012f9e70a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.937564 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.937537    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w777t\" (UniqueName: \"kubernetes.io/projected/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kube-api-access-w777t\") pod \"isvc-sklearn-batcher-predictor-7c69997b57-ktg6q\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:32.983429 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:32.983405    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:33.110861 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:33.110833    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"]
May 20 15:42:33.113876 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:42:33.113849    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918bdba5_698f_4a29_8ff8_ee0012f9e70a.slice/crio-d88e68a247338dc7b01c5ebc65434e081c9516ee16b2bc6f556d6e13feefbd23 WatchSource:0}: Error finding container d88e68a247338dc7b01c5ebc65434e081c9516ee16b2bc6f556d6e13feefbd23: Status 404 returned error can't find the container with id d88e68a247338dc7b01c5ebc65434e081c9516ee16b2bc6f556d6e13feefbd23
May 20 15:42:33.485092 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:33.485060    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerStarted","Data":"d88e68a247338dc7b01c5ebc65434e081c9516ee16b2bc6f556d6e13feefbd23"}
May 20 15:42:37.496793 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:37.496759    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerStarted","Data":"7a75d109948e81e3ca3a719d95ccafc5928bd14a9140ff2890cea204e5edb5fa"}
May 20 15:42:40.506534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:40.506497    2556 generic.go:358] "Generic (PLEG): container finished" podID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerID="7a75d109948e81e3ca3a719d95ccafc5928bd14a9140ff2890cea204e5edb5fa" exitCode=0
May 20 15:42:40.506900 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:40.506578    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerDied","Data":"7a75d109948e81e3ca3a719d95ccafc5928bd14a9140ff2890cea204e5edb5fa"}
May 20 15:42:53.552235 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:53.552199    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerStarted","Data":"fd0ffc7dac12fe3fec15e745ff4eaddb09940b8b62dabbb74523e88d0ecc1922"}
May 20 15:42:55.560203 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:55.560139    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerStarted","Data":"4a43c7b8af20a9f59c632135262716197be83f566ab0a6d072f141bd67ebe356"}
May 20 15:42:58.571106 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:58.571073    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerStarted","Data":"b1496d28271ac30d30802730ae93d36d56aec4e3d18f51f4976093f161e29f51"}
May 20 15:42:58.571492 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:58.571345    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:58.571492 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:58.571460    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:58.572823 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:58.572798    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:42:58.597135 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:58.597060    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podStartSLOduration=1.5105735340000002 podStartE2EDuration="26.597045803s" podCreationTimestamp="2026-05-20 15:42:32 +0000 UTC" firstStartedPulling="2026-05-20 15:42:33.115732618 +0000 UTC m=+595.806988537" lastFinishedPulling="2026-05-20 15:42:58.202204889 +0000 UTC m=+620.893460806" observedRunningTime="2026-05-20 15:42:58.594869131 +0000 UTC m=+621.286125072" watchObservedRunningTime="2026-05-20 15:42:58.597045803 +0000 UTC m=+621.288301732"
May 20 15:42:59.574338 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:59.574308    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:42:59.574777 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:59.574336    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:42:59.575341 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:42:59.575314    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:43:00.577094 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:00.577050    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:43:00.577602 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:00.577576    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:43:00.580962 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:00.580942    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:43:01.580499 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:01.580455    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:43:01.580880 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:01.580750    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:43:11.580824 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:11.580770    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:43:11.581283 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:11.581260    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:43:21.580359 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:21.580302    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:43:21.580771 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:21.580664    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:43:31.581329 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:31.581277    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:43:31.581869 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:31.581784    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:43:41.585817 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:41.585622    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:43:41.586266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:41.586242    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:43:51.581233 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:51.581176    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:43:51.581793 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:43:51.581757    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:01.581918 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:01.581876    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:44:01.582419 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:01.582071    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:44:07.803357 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.803317    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"]
May 20 15:44:07.803932 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.803782    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" containerID="cri-o://fd0ffc7dac12fe3fec15e745ff4eaddb09940b8b62dabbb74523e88d0ecc1922" gracePeriod=30
May 20 15:44:07.803932 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.803807    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" containerID="cri-o://b1496d28271ac30d30802730ae93d36d56aec4e3d18f51f4976093f161e29f51" gracePeriod=30
May 20 15:44:07.803932 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.803808    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy" containerID="cri-o://4a43c7b8af20a9f59c632135262716197be83f566ab0a6d072f141bd67ebe356" gracePeriod=30
May 20 15:44:07.914685 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.914658    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"]
May 20 15:44:07.918139 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.918115    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:07.920436 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.920412    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
May 20 15:44:07.920559 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.920521    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
May 20 15:44:07.935589 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.935562    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"]
May 20 15:44:07.984875 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.984834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:07.985049 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.984900    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmgb\" (UniqueName: \"kubernetes.io/projected/c367218f-f5c9-40c8-ab77-6201aeb7c754-kube-api-access-vcmgb\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:07.986872 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.984954    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c367218f-f5c9-40c8-ab77-6201aeb7c754-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:07.986872 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:07.985504    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c367218f-f5c9-40c8-ab77-6201aeb7c754-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.086892 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.086808    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c367218f-f5c9-40c8-ab77-6201aeb7c754-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.086892 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.086857    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c367218f-f5c9-40c8-ab77-6201aeb7c754-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.087135 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.086904    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.087135 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.086938    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmgb\" (UniqueName: \"kubernetes.io/projected/c367218f-f5c9-40c8-ab77-6201aeb7c754-kube-api-access-vcmgb\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.087135 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:44:08.087088    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
May 20 15:44:08.087279 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:44:08.087189    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls podName:c367218f-f5c9-40c8-ab77-6201aeb7c754 nodeName:}" failed. No retries permitted until 2026-05-20 15:44:08.587165266 +0000 UTC m=+691.278421173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" (UID: "c367218f-f5c9-40c8-ab77-6201aeb7c754") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
May 20 15:44:08.087279 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.087266    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c367218f-f5c9-40c8-ab77-6201aeb7c754-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.087625 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.087605    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c367218f-f5c9-40c8-ab77-6201aeb7c754-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.101242 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.101219    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmgb\" (UniqueName: \"kubernetes.io/projected/c367218f-f5c9-40c8-ab77-6201aeb7c754-kube-api-access-vcmgb\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.590181 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.590144    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.592520 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.592491    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.764432 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.764399    2556 generic.go:358] "Generic (PLEG): container finished" podID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerID="4a43c7b8af20a9f59c632135262716197be83f566ab0a6d072f141bd67ebe356" exitCode=2
May 20 15:44:08.764588 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.764459    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerDied","Data":"4a43c7b8af20a9f59c632135262716197be83f566ab0a6d072f141bd67ebe356"}
May 20 15:44:08.828801 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.828754    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:08.967011 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:08.966981    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"]
May 20 15:44:08.970327 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:44:08.970287    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc367218f_f5c9_40c8_ab77_6201aeb7c754.slice/crio-90383930a6856264389a499351021ff77619727a9cbdb76809da8d1fca07072f WatchSource:0}: Error finding container 90383930a6856264389a499351021ff77619727a9cbdb76809da8d1fca07072f: Status 404 returned error can't find the container with id 90383930a6856264389a499351021ff77619727a9cbdb76809da8d1fca07072f
May 20 15:44:09.768574 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:09.768536    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerStarted","Data":"79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64"}
May 20 15:44:09.768574 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:09.768575    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerStarted","Data":"90383930a6856264389a499351021ff77619727a9cbdb76809da8d1fca07072f"}
May 20 15:44:10.577936 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:10.577899    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
May 20 15:44:11.580524 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:11.580480    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:44:11.580958 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:11.580843    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:12.778406 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:12.778375    2556 generic.go:358] "Generic (PLEG): container finished" podID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerID="fd0ffc7dac12fe3fec15e745ff4eaddb09940b8b62dabbb74523e88d0ecc1922" exitCode=0
May 20 15:44:12.778797 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:12.778442    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerDied","Data":"fd0ffc7dac12fe3fec15e745ff4eaddb09940b8b62dabbb74523e88d0ecc1922"}
May 20 15:44:12.779692 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:12.779671    2556 generic.go:358] "Generic (PLEG): container finished" podID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerID="79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64" exitCode=0
May 20 15:44:12.779797 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:12.779703    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerDied","Data":"79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64"}
May 20 15:44:13.785323 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:13.785289    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerStarted","Data":"722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c"}
May 20 15:44:13.785323 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:13.785330    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerStarted","Data":"391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446"}
May 20 15:44:13.785745 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:13.785339    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerStarted","Data":"08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5"}
May 20 15:44:13.785745 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:13.785633    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:13.785811 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:13.785752    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:13.787026 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:13.786997    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:44:13.827722 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:13.827670    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podStartSLOduration=6.827655397 podStartE2EDuration="6.827655397s" podCreationTimestamp="2026-05-20 15:44:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:44:13.826634053 +0000 UTC m=+696.517889982" watchObservedRunningTime="2026-05-20 15:44:13.827655397 +0000 UTC m=+696.518911326"
May 20 15:44:14.787914 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:14.787879    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:14.788396 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:14.787992    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:44:14.788936 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:14.788913    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:15.578340 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:15.578297    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
May 20 15:44:15.791122 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:15.791066    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:44:15.791568 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:15.791549    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:20.577975 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:20.577927    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
May 20 15:44:20.578462 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:20.578085    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:44:20.796213 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:20.796186    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:44:20.796794 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:20.796767    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:44:20.797017 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:20.796992    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:21.580887 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:21.580847    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:44:21.581305 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:21.581186    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:25.578076 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:25.578029    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
May 20 15:44:30.577900 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:30.577819    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
May 20 15:44:30.797367 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:30.797328    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:44:30.797826 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:30.797801    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:31.581302 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:31.581253    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
May 20 15:44:31.581746 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:31.581431    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:44:31.581746 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:31.581651    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:31.581830 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:31.581791    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:44:35.578040 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:35.577994    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
May 20 15:44:37.850436 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:37.850406    2556 generic.go:358] "Generic (PLEG): container finished" podID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerID="b1496d28271ac30d30802730ae93d36d56aec4e3d18f51f4976093f161e29f51" exitCode=0
May 20 15:44:37.850739 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:37.850491    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerDied","Data":"b1496d28271ac30d30802730ae93d36d56aec4e3d18f51f4976093f161e29f51"}
May 20 15:44:37.952659 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:37.952636    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:44:38.109336 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.109261    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kserve-provision-location\") pod \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") "
May 20 15:44:38.109336 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.109319    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918bdba5-698f-4a29-8ff8-ee0012f9e70a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") "
May 20 15:44:38.109553 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.109346    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w777t\" (UniqueName: \"kubernetes.io/projected/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kube-api-access-w777t\") pod \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") "
May 20 15:44:38.109553 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.109376    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918bdba5-698f-4a29-8ff8-ee0012f9e70a-proxy-tls\") pod \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\" (UID: \"918bdba5-698f-4a29-8ff8-ee0012f9e70a\") "
May 20 15:44:38.109662 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.109583    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "918bdba5-698f-4a29-8ff8-ee0012f9e70a" (UID: "918bdba5-698f-4a29-8ff8-ee0012f9e70a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:44:38.109735 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.109708    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918bdba5-698f-4a29-8ff8-ee0012f9e70a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "918bdba5-698f-4a29-8ff8-ee0012f9e70a" (UID: "918bdba5-698f-4a29-8ff8-ee0012f9e70a"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:44:38.111447 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.111423    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/918bdba5-698f-4a29-8ff8-ee0012f9e70a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "918bdba5-698f-4a29-8ff8-ee0012f9e70a" (UID: "918bdba5-698f-4a29-8ff8-ee0012f9e70a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:44:38.111565 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.111538    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kube-api-access-w777t" (OuterVolumeSpecName: "kube-api-access-w777t") pod "918bdba5-698f-4a29-8ff8-ee0012f9e70a" (UID: "918bdba5-698f-4a29-8ff8-ee0012f9e70a"). InnerVolumeSpecName "kube-api-access-w777t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:44:38.209805 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.209775    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918bdba5-698f-4a29-8ff8-ee0012f9e70a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:44:38.209805 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.209797    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w777t\" (UniqueName: \"kubernetes.io/projected/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kube-api-access-w777t\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:44:38.209805 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.209807    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918bdba5-698f-4a29-8ff8-ee0012f9e70a-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:44:38.210018 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.209816    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918bdba5-698f-4a29-8ff8-ee0012f9e70a-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:44:38.855737 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.855704    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q" event={"ID":"918bdba5-698f-4a29-8ff8-ee0012f9e70a","Type":"ContainerDied","Data":"d88e68a247338dc7b01c5ebc65434e081c9516ee16b2bc6f556d6e13feefbd23"}
May 20 15:44:38.856214 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.855750    2556 scope.go:117] "RemoveContainer" containerID="b1496d28271ac30d30802730ae93d36d56aec4e3d18f51f4976093f161e29f51"
May 20 15:44:38.856214 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.855753    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"
May 20 15:44:38.865602 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.865585    2556 scope.go:117] "RemoveContainer" containerID="4a43c7b8af20a9f59c632135262716197be83f566ab0a6d072f141bd67ebe356"
May 20 15:44:38.872075 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.872056    2556 scope.go:117] "RemoveContainer" containerID="fd0ffc7dac12fe3fec15e745ff4eaddb09940b8b62dabbb74523e88d0ecc1922"
May 20 15:44:38.878431 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.878357    2556 scope.go:117] "RemoveContainer" containerID="7a75d109948e81e3ca3a719d95ccafc5928bd14a9140ff2890cea204e5edb5fa"
May 20 15:44:38.879756 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.879736    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"]
May 20 15:44:38.884044 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:38.884026    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7c69997b57-ktg6q"]
May 20 15:44:39.890758 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:39.890727    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" path="/var/lib/kubelet/pods/918bdba5-698f-4a29-8ff8-ee0012f9e70a/volumes"
May 20 15:44:40.797375 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:40.797332    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:44:40.797852 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:40.797821    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:44:50.796772 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:50.796714    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:44:50.797227 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:44:50.797194    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:00.796674 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:00.796636    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:45:00.797136 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:00.796976    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:10.797526 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:10.797459    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:45:10.798041 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:10.797926    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:20.797171 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:20.797138    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:45:20.797575 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:20.797207    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:45:32.945164 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.945132    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"]
May 20 15:45:32.945571 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.945480    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" containerID="cri-o://08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5" gracePeriod=30
May 20 15:45:32.945571 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.945525    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" containerID="cri-o://722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c" gracePeriod=30
May 20 15:45:32.945571 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.945553    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy" containerID="cri-o://391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446" gracePeriod=30
May 20 15:45:32.997301 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997277    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"]
May 20 15:45:32.997537 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997525    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent"
May 20 15:45:32.997591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997539    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent"
May 20 15:45:32.997591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997550    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="storage-initializer"
May 20 15:45:32.997591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997557    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="storage-initializer"
May 20 15:45:32.997591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997565    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container"
May 20 15:45:32.997591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997570    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container"
May 20 15:45:32.997591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997584    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy"
May 20 15:45:32.997591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997589    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy"
May 20 15:45:32.997812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997628    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kserve-container"
May 20 15:45:32.997812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997635    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="kube-rbac-proxy"
May 20 15:45:32.997812 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:32.997641    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="918bdba5-698f-4a29-8ff8-ee0012f9e70a" containerName="agent"
May 20 15:45:33.000310 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.000296    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.002895 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.002877    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
May 20 15:45:33.002964 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.002877    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
May 20 15:45:33.010003 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.009981    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"]
May 20 15:45:33.102642 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.102622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.102727 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.102668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljggs\" (UniqueName: \"kubernetes.io/projected/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-kube-api-access-ljggs\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.102727 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.102706    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.203344 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.203279    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ljggs\" (UniqueName: \"kubernetes.io/projected/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-kube-api-access-ljggs\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.203344 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.203333    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.203474 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.203397    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.203510 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:45:33.203484    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
May 20 15:45:33.203546 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:45:33.203538    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls podName:b5fcc889-3bf1-46bc-bdbd-bdcc73f16461 nodeName:}" failed. No retries permitted until 2026-05-20 15:45:33.703522301 +0000 UTC m=+776.394778207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls") pod "message-dumper-predictor-c7d86bcbd-xbzv8" (UID: "b5fcc889-3bf1-46bc-bdbd-bdcc73f16461") : secret "message-dumper-predictor-serving-cert" not found
May 20 15:45:33.204163 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.204141    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.212556 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.212533    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljggs\" (UniqueName: \"kubernetes.io/projected/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-kube-api-access-ljggs\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.707521 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.707484    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.709702 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.709675    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-xbzv8\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:33.909210 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:33.909169    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:34.001226 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:34.001197    2556 generic.go:358] "Generic (PLEG): container finished" podID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerID="391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446" exitCode=2
May 20 15:45:34.001538 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:34.001253    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerDied","Data":"391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446"}
May 20 15:45:34.034265 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:34.034235    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"]
May 20 15:45:34.037644 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:45:34.037618    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fcc889_3bf1_46bc_bdbd_bdcc73f16461.slice/crio-cac0c61f1d0db21f7c85735d594a3332117d0923280c39f3ddc755d7d0b177cb WatchSource:0}: Error finding container cac0c61f1d0db21f7c85735d594a3332117d0923280c39f3ddc755d7d0b177cb: Status 404 returned error can't find the container with id cac0c61f1d0db21f7c85735d594a3332117d0923280c39f3ddc755d7d0b177cb
May 20 15:45:34.039282 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:34.039265    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:45:35.006011 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:35.005971    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" event={"ID":"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461","Type":"ContainerStarted","Data":"cac0c61f1d0db21f7c85735d594a3332117d0923280c39f3ddc755d7d0b177cb"}
May 20 15:45:35.791404 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:35.791355    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
May 20 15:45:36.011465 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:36.011432    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" event={"ID":"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461","Type":"ContainerStarted","Data":"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba"}
May 20 15:45:36.011465 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:36.011472    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" event={"ID":"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461","Type":"ContainerStarted","Data":"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6"}
May 20 15:45:36.011952 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:36.011617    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:36.030320 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:36.030276    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" podStartSLOduration=2.9837968740000003 podStartE2EDuration="4.030263487s" podCreationTimestamp="2026-05-20 15:45:32 +0000 UTC" firstStartedPulling="2026-05-20 15:45:34.03938826 +0000 UTC m=+776.730644169" lastFinishedPulling="2026-05-20 15:45:35.085854868 +0000 UTC m=+777.777110782" observedRunningTime="2026-05-20 15:45:36.028318542 +0000 UTC m=+778.719574470" watchObservedRunningTime="2026-05-20 15:45:36.030263487 +0000 UTC m=+778.721519410"
May 20 15:45:37.016349 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:37.016316    2556 generic.go:358] "Generic (PLEG): container finished" podID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerID="08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5" exitCode=0
May 20 15:45:37.016698 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:37.016389    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerDied","Data":"08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5"}
May 20 15:45:37.016698 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:37.016682    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:37.018267 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:37.018247    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:40.791548 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:40.791509    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
May 20 15:45:40.796854 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:40.796829    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:45:40.797144 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:40.797119    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:44.025277 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:44.025245    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:45:45.792171 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:45.792127    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
May 20 15:45:45.792675 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:45.792306    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:45:50.791648 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:50.791600    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
May 20 15:45:50.797116 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:50.797062    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:45:50.797494 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:50.797472    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:45:53.074906 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.074868    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"]
May 20 15:45:53.082747 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.082725    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.085355 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.085326    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
May 20 15:45:53.085483 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.085333    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
May 20 15:45:53.089140 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.089113    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"]
May 20 15:45:53.245638 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.245608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4t9\" (UniqueName: \"kubernetes.io/projected/037d7570-b63a-448b-81f9-52f6bdb24e9a-kube-api-access-dv4t9\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.245755 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.245665    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037d7570-b63a-448b-81f9-52f6bdb24e9a-proxy-tls\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.245755 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.245712    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037d7570-b63a-448b-81f9-52f6bdb24e9a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.245755 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.245734    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037d7570-b63a-448b-81f9-52f6bdb24e9a-kserve-provision-location\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.346789 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.346735    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037d7570-b63a-448b-81f9-52f6bdb24e9a-proxy-tls\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.346789 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.346768    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037d7570-b63a-448b-81f9-52f6bdb24e9a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.346789 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.346789    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037d7570-b63a-448b-81f9-52f6bdb24e9a-kserve-provision-location\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.346976 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.346818    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4t9\" (UniqueName: \"kubernetes.io/projected/037d7570-b63a-448b-81f9-52f6bdb24e9a-kube-api-access-dv4t9\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.347228 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.347198    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037d7570-b63a-448b-81f9-52f6bdb24e9a-kserve-provision-location\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.347491 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.347473    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037d7570-b63a-448b-81f9-52f6bdb24e9a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.349315 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.349294    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037d7570-b63a-448b-81f9-52f6bdb24e9a-proxy-tls\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.355658 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.355639    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4t9\" (UniqueName: \"kubernetes.io/projected/037d7570-b63a-448b-81f9-52f6bdb24e9a-kube-api-access-dv4t9\") pod \"isvc-logger-predictor-f47486ddd-tjvgs\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.393925 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.393903    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:53.519026 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:53.518992    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"]
May 20 15:45:53.522082 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:45:53.522054    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037d7570_b63a_448b_81f9_52f6bdb24e9a.slice/crio-0bf1e606451911582152fc9d6c12e4531edcf77045111437ca47ea75565a61e3 WatchSource:0}: Error finding container 0bf1e606451911582152fc9d6c12e4531edcf77045111437ca47ea75565a61e3: Status 404 returned error can't find the container with id 0bf1e606451911582152fc9d6c12e4531edcf77045111437ca47ea75565a61e3
May 20 15:45:54.064294 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:54.064255    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerStarted","Data":"7079aa2b6f62f411ad1686f6327165f5d65d50af3174009660ee9cda8ada83cf"}
May 20 15:45:54.064434 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:54.064299    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerStarted","Data":"0bf1e606451911582152fc9d6c12e4531edcf77045111437ca47ea75565a61e3"}
May 20 15:45:55.791905 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:55.791872    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
May 20 15:45:58.076136 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:58.076085    2556 generic.go:358] "Generic (PLEG): container finished" podID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerID="7079aa2b6f62f411ad1686f6327165f5d65d50af3174009660ee9cda8ada83cf" exitCode=0
May 20 15:45:58.076472 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:58.076161    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerDied","Data":"7079aa2b6f62f411ad1686f6327165f5d65d50af3174009660ee9cda8ada83cf"}
May 20 15:45:59.086012 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:59.085975    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerStarted","Data":"28e3fb0553659bc4519982df73301c69236287ad65354bdd39a2fe7fad6ae70c"}
May 20 15:45:59.086413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:59.086020    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerStarted","Data":"b37fb70e9fb822bb67cae1d33a269c80f7fd727f132bb6a2a1b845b5f0ab3f67"}
May 20 15:45:59.086413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:59.086035    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerStarted","Data":"a7368781e8377934d34ea2e39d1701b58d0c3a9b8f8999defc28da8f4374961b"}
May 20 15:45:59.086413 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:59.086366    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:59.086558 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:59.086460    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:45:59.087463 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:59.087433    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:45:59.107210 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:45:59.107169    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podStartSLOduration=6.107156763 podStartE2EDuration="6.107156763s" podCreationTimestamp="2026-05-20 15:45:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:45:59.10669774 +0000 UTC m=+801.797953705" watchObservedRunningTime="2026-05-20 15:45:59.107156763 +0000 UTC m=+801.798412690"
May 20 15:46:00.089072 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:00.089037    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:46:00.089551 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:00.089221    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:46:00.090081 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:00.090057    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:00.792022 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:00.791984    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
May 20 15:46:00.797435 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:00.797383    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:5000: connect: connection refused"
May 20 15:46:00.797594 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:00.797549    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:46:00.797771 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:00.797746    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:00.797877 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:00.797863    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:46:01.092215 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:01.092142    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:46:01.092573 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:01.092539    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:03.086893 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.086871    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:46:03.097838 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.097814    2556 generic.go:358] "Generic (PLEG): container finished" podID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerID="722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c" exitCode=0
May 20 15:46:03.097937 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.097888    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerDied","Data":"722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c"}
May 20 15:46:03.097937 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.097907    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"
May 20 15:46:03.097937 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.097921    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z" event={"ID":"c367218f-f5c9-40c8-ab77-6201aeb7c754","Type":"ContainerDied","Data":"90383930a6856264389a499351021ff77619727a9cbdb76809da8d1fca07072f"}
May 20 15:46:03.098045 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.097941    2556 scope.go:117] "RemoveContainer" containerID="722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c"
May 20 15:46:03.104756 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.104738    2556 scope.go:117] "RemoveContainer" containerID="391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446"
May 20 15:46:03.112667 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.112639    2556 scope.go:117] "RemoveContainer" containerID="08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5"
May 20 15:46:03.121012 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.120995    2556 scope.go:117] "RemoveContainer" containerID="79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64"
May 20 15:46:03.127180 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.127161    2556 scope.go:117] "RemoveContainer" containerID="722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c"
May 20 15:46:03.127406 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:46:03.127388    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c\": container with ID starting with 722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c not found: ID does not exist" containerID="722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c"
May 20 15:46:03.127453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.127414    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c"} err="failed to get container status \"722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c\": rpc error: code = NotFound desc = could not find container \"722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c\": container with ID starting with 722f6c79524279c0e96831b956224530cbf28f36c3a223624fda06690fb82a0c not found: ID does not exist"
May 20 15:46:03.127453 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.127432    2556 scope.go:117] "RemoveContainer" containerID="391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446"
May 20 15:46:03.127639 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:46:03.127623    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446\": container with ID starting with 391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446 not found: ID does not exist" containerID="391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446"
May 20 15:46:03.127682 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.127646    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446"} err="failed to get container status \"391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446\": rpc error: code = NotFound desc = could not find container \"391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446\": container with ID starting with 391e3af2c8161c6a9891da3c2d3d79c1d3b7f2234908b5cf119e7ce19a149446 not found: ID does not exist"
May 20 15:46:03.127682 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.127662    2556 scope.go:117] "RemoveContainer" containerID="08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5"
May 20 15:46:03.127896 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:46:03.127877    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5\": container with ID starting with 08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5 not found: ID does not exist" containerID="08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5"
May 20 15:46:03.127959 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.127907    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5"} err="failed to get container status \"08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5\": rpc error: code = NotFound desc = could not find container \"08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5\": container with ID starting with 08f64c86f372780f6e13d69389572c97773d0b943e1aaadac9a690e1aa3cb3c5 not found: ID does not exist"
May 20 15:46:03.127959 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.127932    2556 scope.go:117] "RemoveContainer" containerID="79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64"
May 20 15:46:03.128177 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:46:03.128157    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64\": container with ID starting with 79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64 not found: ID does not exist" containerID="79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64"
May 20 15:46:03.128229 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.128185    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64"} err="failed to get container status \"79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64\": rpc error: code = NotFound desc = could not find container \"79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64\": container with ID starting with 79ce3f01b5c66d35e993c0c57be7f28a4d671701419f9530cc34ea0cda58ab64 not found: ID does not exist"
May 20 15:46:03.215468 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.215386    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcmgb\" (UniqueName: \"kubernetes.io/projected/c367218f-f5c9-40c8-ab77-6201aeb7c754-kube-api-access-vcmgb\") pod \"c367218f-f5c9-40c8-ab77-6201aeb7c754\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") "
May 20 15:46:03.215468 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.215425    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c367218f-f5c9-40c8-ab77-6201aeb7c754-kserve-provision-location\") pod \"c367218f-f5c9-40c8-ab77-6201aeb7c754\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") "
May 20 15:46:03.215468 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.215458    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c367218f-f5c9-40c8-ab77-6201aeb7c754-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"c367218f-f5c9-40c8-ab77-6201aeb7c754\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") "
May 20 15:46:03.215719 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.215509    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls\") pod \"c367218f-f5c9-40c8-ab77-6201aeb7c754\" (UID: \"c367218f-f5c9-40c8-ab77-6201aeb7c754\") "
May 20 15:46:03.215852 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.215828    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c367218f-f5c9-40c8-ab77-6201aeb7c754-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c367218f-f5c9-40c8-ab77-6201aeb7c754" (UID: "c367218f-f5c9-40c8-ab77-6201aeb7c754"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:46:03.215852 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.215839    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c367218f-f5c9-40c8-ab77-6201aeb7c754-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "c367218f-f5c9-40c8-ab77-6201aeb7c754" (UID: "c367218f-f5c9-40c8-ab77-6201aeb7c754"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:46:03.217461 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.217440    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c367218f-f5c9-40c8-ab77-6201aeb7c754" (UID: "c367218f-f5c9-40c8-ab77-6201aeb7c754"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:46:03.217602 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.217585    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c367218f-f5c9-40c8-ab77-6201aeb7c754-kube-api-access-vcmgb" (OuterVolumeSpecName: "kube-api-access-vcmgb") pod "c367218f-f5c9-40c8-ab77-6201aeb7c754" (UID: "c367218f-f5c9-40c8-ab77-6201aeb7c754"). InnerVolumeSpecName "kube-api-access-vcmgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:46:03.316444 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.316421    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmgb\" (UniqueName: \"kubernetes.io/projected/c367218f-f5c9-40c8-ab77-6201aeb7c754-kube-api-access-vcmgb\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:46:03.316444 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.316440    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c367218f-f5c9-40c8-ab77-6201aeb7c754-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:46:03.316574 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.316451    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c367218f-f5c9-40c8-ab77-6201aeb7c754-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:46:03.316574 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.316460    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c367218f-f5c9-40c8-ab77-6201aeb7c754-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:46:03.422454 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.422427    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"]
May 20 15:46:03.427249 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.427225    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dc5b85976-9jt5z"]
May 20 15:46:03.891111 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:03.891077    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" path="/var/lib/kubelet/pods/c367218f-f5c9-40c8-ab77-6201aeb7c754/volumes"
May 20 15:46:06.096539 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:06.096499    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:46:06.097119 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:06.097067    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:46:06.097557 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:06.097534    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:16.097756 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:16.097711    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:46:16.098241 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:16.098150    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:26.097723 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:26.097679    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:46:26.098159 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:26.098127    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:36.097596 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:36.097549    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:46:36.098036 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:36.098011    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:46.097193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:46.097157    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:46:46.097646 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:46.097617    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:46:56.097597 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:56.097559    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:46:56.098125 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:46:56.098033    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:47:06.098028 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:06.097994    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:47:06.098517 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:06.098457    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:47:18.091030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.090998    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-xbzv8_b5fcc889-3bf1-46bc-bdbd-bdcc73f16461/kserve-container/0.log"
May 20 15:47:18.272058 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.272027    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"]
May 20 15:47:18.272430 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.272367    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerName="kserve-container" containerID="cri-o://d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6" gracePeriod=30
May 20 15:47:18.272587 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.272408    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerName="kube-rbac-proxy" containerID="cri-o://4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba" gracePeriod=30
May 20 15:47:18.348552 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.348468    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"]
May 20 15:47:18.348830 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.348801    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" containerID="cri-o://a7368781e8377934d34ea2e39d1701b58d0c3a9b8f8999defc28da8f4374961b" gracePeriod=30
May 20 15:47:18.348972 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.348842    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy" containerID="cri-o://b37fb70e9fb822bb67cae1d33a269c80f7fd727f132bb6a2a1b845b5f0ab3f67" gracePeriod=30
May 20 15:47:18.348972 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.348825    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" containerID="cri-o://28e3fb0553659bc4519982df73301c69236287ad65354bdd39a2fe7fad6ae70c" gracePeriod=30
May 20 15:47:18.383212 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383191    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"]
May 20 15:47:18.383475 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383462    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy"
May 20 15:47:18.383517 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383477    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy"
May 20 15:47:18.383517 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383488    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent"
May 20 15:47:18.383517 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383493    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent"
May 20 15:47:18.383517 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383500    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="storage-initializer"
May 20 15:47:18.383517 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383506    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="storage-initializer"
May 20 15:47:18.383517 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383517    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container"
May 20 15:47:18.383688 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383523    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container"
May 20 15:47:18.383688 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383561    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kserve-container"
May 20 15:47:18.383688 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383571    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="kube-rbac-proxy"
May 20 15:47:18.383688 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.383578    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c367218f-f5c9-40c8-ab77-6201aeb7c754" containerName="agent"
May 20 15:47:18.386264 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.386250    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.388768 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.388749    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
May 20 15:47:18.388865 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.388749    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
May 20 15:47:18.399877 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.399857    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"]
May 20 15:47:18.466411 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.466383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de721d79-c885-4a21-a554-1863c81cdc52-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.466518 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.466429    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de721d79-c885-4a21-a554-1863c81cdc52-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.466518 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.466484    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de721d79-c885-4a21-a554-1863c81cdc52-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.466623 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.466541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfwv\" (UniqueName: \"kubernetes.io/projected/de721d79-c885-4a21-a554-1863c81cdc52-kube-api-access-kkfwv\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.567855 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.567830    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de721d79-c885-4a21-a554-1863c81cdc52-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.567966 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.567864    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de721d79-c885-4a21-a554-1863c81cdc52-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.567966 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.567898    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de721d79-c885-4a21-a554-1863c81cdc52-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.568073 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.567957    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfwv\" (UniqueName: \"kubernetes.io/projected/de721d79-c885-4a21-a554-1863c81cdc52-kube-api-access-kkfwv\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.568269 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.568250    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de721d79-c885-4a21-a554-1863c81cdc52-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.568530 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.568510    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de721d79-c885-4a21-a554-1863c81cdc52-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.570247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.570227    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de721d79-c885-4a21-a554-1863c81cdc52-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.575876 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.575861    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:47:18.577242 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.577225    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfwv\" (UniqueName: \"kubernetes.io/projected/de721d79-c885-4a21-a554-1863c81cdc52-kube-api-access-kkfwv\") pod \"isvc-lightgbm-predictor-bdf964bd-2mztd\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.669205 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.669129    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-message-dumper-kube-rbac-proxy-sar-config\") pod \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") "
May 20 15:47:18.669205 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.669167    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls\") pod \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") "
May 20 15:47:18.669205 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.669203    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljggs\" (UniqueName: \"kubernetes.io/projected/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-kube-api-access-ljggs\") pod \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\" (UID: \"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461\") "
May 20 15:47:18.669524 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.669505    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" (UID: "b5fcc889-3bf1-46bc-bdbd-bdcc73f16461"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:47:18.671182 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.671162    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" (UID: "b5fcc889-3bf1-46bc-bdbd-bdcc73f16461"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:47:18.671285 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.671266    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-kube-api-access-ljggs" (OuterVolumeSpecName: "kube-api-access-ljggs") pod "b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" (UID: "b5fcc889-3bf1-46bc-bdbd-bdcc73f16461"). InnerVolumeSpecName "kube-api-access-ljggs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:47:18.695440 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.695416    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:18.770018 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.769994    2556 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:47:18.770018 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.770019    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:47:18.770189 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.770030    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljggs\" (UniqueName: \"kubernetes.io/projected/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461-kube-api-access-ljggs\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:47:18.824824 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:18.824797    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"]
May 20 15:47:18.828214 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:47:18.828178    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde721d79_c885_4a21_a554_1863c81cdc52.slice/crio-7546cb8746bed9d7faa2f23672a1878a33f16ec823c53f145bba64fd96f00bfc WatchSource:0}: Error finding container 7546cb8746bed9d7faa2f23672a1878a33f16ec823c53f145bba64fd96f00bfc: Status 404 returned error can't find the container with id 7546cb8746bed9d7faa2f23672a1878a33f16ec823c53f145bba64fd96f00bfc
May 20 15:47:19.308193 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.308160    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" event={"ID":"de721d79-c885-4a21-a554-1863c81cdc52","Type":"ContainerStarted","Data":"8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69"}
May 20 15:47:19.308632 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.308200    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" event={"ID":"de721d79-c885-4a21-a554-1863c81cdc52","Type":"ContainerStarted","Data":"7546cb8746bed9d7faa2f23672a1878a33f16ec823c53f145bba64fd96f00bfc"}
May 20 15:47:19.310513 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.310479    2556 generic.go:358] "Generic (PLEG): container finished" podID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerID="b37fb70e9fb822bb67cae1d33a269c80f7fd727f132bb6a2a1b845b5f0ab3f67" exitCode=2
May 20 15:47:19.310655 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.310544    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerDied","Data":"b37fb70e9fb822bb67cae1d33a269c80f7fd727f132bb6a2a1b845b5f0ab3f67"}
May 20 15:47:19.312027 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.312001    2556 generic.go:358] "Generic (PLEG): container finished" podID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerID="4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba" exitCode=2
May 20 15:47:19.312027 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.312027    2556 generic.go:358] "Generic (PLEG): container finished" podID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerID="d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6" exitCode=2
May 20 15:47:19.312200 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.312082    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" event={"ID":"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461","Type":"ContainerDied","Data":"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba"}
May 20 15:47:19.312200 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.312088    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"
May 20 15:47:19.312200 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.312137    2556 scope.go:117] "RemoveContainer" containerID="4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba"
May 20 15:47:19.312360 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.312126    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" event={"ID":"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461","Type":"ContainerDied","Data":"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6"}
May 20 15:47:19.312360 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.312297    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8" event={"ID":"b5fcc889-3bf1-46bc-bdbd-bdcc73f16461","Type":"ContainerDied","Data":"cac0c61f1d0db21f7c85735d594a3332117d0923280c39f3ddc755d7d0b177cb"}
May 20 15:47:19.319930 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.319851    2556 scope.go:117] "RemoveContainer" containerID="d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6"
May 20 15:47:19.326711 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.326689    2556 scope.go:117] "RemoveContainer" containerID="4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba"
May 20 15:47:19.326969 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:47:19.326951    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba\": container with ID starting with 4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba not found: ID does not exist" containerID="4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba"
May 20 15:47:19.327026 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.326978    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba"} err="failed to get container status \"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba\": rpc error: code = NotFound desc = could not find container \"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba\": container with ID starting with 4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba not found: ID does not exist"
May 20 15:47:19.327026 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.326996    2556 scope.go:117] "RemoveContainer" containerID="d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6"
May 20 15:47:19.327396 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:47:19.327370    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6\": container with ID starting with d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6 not found: ID does not exist" containerID="d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6"
May 20 15:47:19.327479 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.327406    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6"} err="failed to get container status \"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6\": rpc error: code = NotFound desc = could not find container \"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6\": container with ID starting with d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6 not found: ID does not exist"
May 20 15:47:19.327479 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.327429    2556 scope.go:117] "RemoveContainer" containerID="4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba"
May 20 15:47:19.327683 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.327659    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba"} err="failed to get container status \"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba\": rpc error: code = NotFound desc = could not find container \"4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba\": container with ID starting with 4863a983708459d599b27c873b215ced876a59dc306bc7290d03292cf7f450ba not found: ID does not exist"
May 20 15:47:19.327757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.327686    2556 scope.go:117] "RemoveContainer" containerID="d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6"
May 20 15:47:19.327919 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.327896    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6"} err="failed to get container status \"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6\": rpc error: code = NotFound desc = could not find container \"d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6\": container with ID starting with d58b188173c6a5dda99c1e0b4c340d260e5b97a7fc7aa59c454cf9c765f453d6 not found: ID does not exist"
May 20 15:47:19.343826 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.343801    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"]
May 20 15:47:19.350010 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.349980    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-xbzv8"]
May 20 15:47:19.890766 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:19.890728    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" path="/var/lib/kubelet/pods/b5fcc889-3bf1-46bc-bdbd-bdcc73f16461/volumes"
May 20 15:47:21.092970 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:21.092923    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
May 20 15:47:22.322686 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:22.322605    2556 generic.go:358] "Generic (PLEG): container finished" podID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerID="a7368781e8377934d34ea2e39d1701b58d0c3a9b8f8999defc28da8f4374961b" exitCode=0
May 20 15:47:22.322686 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:22.322641    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerDied","Data":"a7368781e8377934d34ea2e39d1701b58d0c3a9b8f8999defc28da8f4374961b"}
May 20 15:47:23.326350 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:23.326316    2556 generic.go:358] "Generic (PLEG): container finished" podID="de721d79-c885-4a21-a554-1863c81cdc52" containerID="8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69" exitCode=0
May 20 15:47:23.326786 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:23.326356    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" event={"ID":"de721d79-c885-4a21-a554-1863c81cdc52","Type":"ContainerDied","Data":"8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69"}
May 20 15:47:26.092512 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:26.092471    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
May 20 15:47:26.098309 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:26.098276    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:47:26.098442 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:26.098415    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:47:30.350256 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:30.350220    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" event={"ID":"de721d79-c885-4a21-a554-1863c81cdc52","Type":"ContainerStarted","Data":"317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a"}
May 20 15:47:30.350732 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:30.350266    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" event={"ID":"de721d79-c885-4a21-a554-1863c81cdc52","Type":"ContainerStarted","Data":"8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34"}
May 20 15:47:30.350732 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:30.350473    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:30.372821 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:30.372778    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podStartSLOduration=6.325749278 podStartE2EDuration="12.372766699s" podCreationTimestamp="2026-05-20 15:47:18 +0000 UTC" firstStartedPulling="2026-05-20 15:47:23.327698455 +0000 UTC m=+886.018954361" lastFinishedPulling="2026-05-20 15:47:29.374715876 +0000 UTC m=+892.065971782" observedRunningTime="2026-05-20 15:47:30.371449644 +0000 UTC m=+893.062705585" watchObservedRunningTime="2026-05-20 15:47:30.372766699 +0000 UTC m=+893.064022627"
May 20 15:47:31.092380 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:31.092344    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
May 20 15:47:31.092549 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:31.092479    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:47:31.352905 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:31.352824    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:31.353993 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:31.353965    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:47:32.355833 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:32.355794    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:47:36.092925 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:36.092887    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
May 20 15:47:36.097482 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:36.097454    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:47:36.098665 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:36.098642    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:47:37.360454 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:37.360422    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:47:37.361027 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:37.360994    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:47:41.092352 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:41.092301    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
May 20 15:47:46.092954 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:46.092917    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
May 20 15:47:46.097477 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:46.097450    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:47:46.097594 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:46.097548    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:47:46.098681 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:46.098662    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
May 20 15:47:46.098754 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:46.098744    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:47:47.361053 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:47.361015    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:47:48.399531 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.399499    2556 generic.go:358] "Generic (PLEG): container finished" podID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerID="28e3fb0553659bc4519982df73301c69236287ad65354bdd39a2fe7fad6ae70c" exitCode=0
May 20 15:47:48.399844 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.399552    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerDied","Data":"28e3fb0553659bc4519982df73301c69236287ad65354bdd39a2fe7fad6ae70c"}
May 20 15:47:48.493236 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.493212    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:47:48.592248 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.592223    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037d7570-b63a-448b-81f9-52f6bdb24e9a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"037d7570-b63a-448b-81f9-52f6bdb24e9a\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") "
May 20 15:47:48.592367 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.592259    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037d7570-b63a-448b-81f9-52f6bdb24e9a-kserve-provision-location\") pod \"037d7570-b63a-448b-81f9-52f6bdb24e9a\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") "
May 20 15:47:48.592367 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.592308    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv4t9\" (UniqueName: \"kubernetes.io/projected/037d7570-b63a-448b-81f9-52f6bdb24e9a-kube-api-access-dv4t9\") pod \"037d7570-b63a-448b-81f9-52f6bdb24e9a\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") "
May 20 15:47:48.592367 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.592345    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037d7570-b63a-448b-81f9-52f6bdb24e9a-proxy-tls\") pod \"037d7570-b63a-448b-81f9-52f6bdb24e9a\" (UID: \"037d7570-b63a-448b-81f9-52f6bdb24e9a\") "
May 20 15:47:48.592557 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.592534    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037d7570-b63a-448b-81f9-52f6bdb24e9a-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "037d7570-b63a-448b-81f9-52f6bdb24e9a" (UID: "037d7570-b63a-448b-81f9-52f6bdb24e9a"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:47:48.592650 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.592626    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037d7570-b63a-448b-81f9-52f6bdb24e9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "037d7570-b63a-448b-81f9-52f6bdb24e9a" (UID: "037d7570-b63a-448b-81f9-52f6bdb24e9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:47:48.594463 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.594437    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037d7570-b63a-448b-81f9-52f6bdb24e9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "037d7570-b63a-448b-81f9-52f6bdb24e9a" (UID: "037d7570-b63a-448b-81f9-52f6bdb24e9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:47:48.594572 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.594496    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037d7570-b63a-448b-81f9-52f6bdb24e9a-kube-api-access-dv4t9" (OuterVolumeSpecName: "kube-api-access-dv4t9") pod "037d7570-b63a-448b-81f9-52f6bdb24e9a" (UID: "037d7570-b63a-448b-81f9-52f6bdb24e9a"). InnerVolumeSpecName "kube-api-access-dv4t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:47:48.693683 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.693617    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037d7570-b63a-448b-81f9-52f6bdb24e9a-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:47:48.693683 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.693651    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037d7570-b63a-448b-81f9-52f6bdb24e9a-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:47:48.693683 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.693662    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037d7570-b63a-448b-81f9-52f6bdb24e9a-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:47:48.693683 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:48.693672    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv4t9\" (UniqueName: \"kubernetes.io/projected/037d7570-b63a-448b-81f9-52f6bdb24e9a-kube-api-access-dv4t9\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:47:49.404290 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.404254    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs" event={"ID":"037d7570-b63a-448b-81f9-52f6bdb24e9a","Type":"ContainerDied","Data":"0bf1e606451911582152fc9d6c12e4531edcf77045111437ca47ea75565a61e3"}
May 20 15:47:49.404751 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.404300    2556 scope.go:117] "RemoveContainer" containerID="28e3fb0553659bc4519982df73301c69236287ad65354bdd39a2fe7fad6ae70c"
May 20 15:47:49.404751 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.404326    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"
May 20 15:47:49.412985 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.412968    2556 scope.go:117] "RemoveContainer" containerID="b37fb70e9fb822bb67cae1d33a269c80f7fd727f132bb6a2a1b845b5f0ab3f67"
May 20 15:47:49.419521 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.419505    2556 scope.go:117] "RemoveContainer" containerID="a7368781e8377934d34ea2e39d1701b58d0c3a9b8f8999defc28da8f4374961b"
May 20 15:47:49.425662 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.425633    2556 scope.go:117] "RemoveContainer" containerID="7079aa2b6f62f411ad1686f6327165f5d65d50af3174009660ee9cda8ada83cf"
May 20 15:47:49.428503 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.428483    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"]
May 20 15:47:49.432419 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.432397    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-f47486ddd-tjvgs"]
May 20 15:47:49.890355 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:49.890328    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" path="/var/lib/kubelet/pods/037d7570-b63a-448b-81f9-52f6bdb24e9a/volumes"
May 20 15:47:57.361666 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:47:57.361633    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:48:07.361195 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:07.361157    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:48:17.361399 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:17.361356    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:48:27.361027 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:27.360978    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:48:37.361157 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:37.361093    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
May 20 15:48:47.361938 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:47.361897    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:48:48.489148 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.489118    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"]
May 20 15:48:48.489554 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.489396    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container" containerID="cri-o://8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34" gracePeriod=30
May 20 15:48:48.489554 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.489540    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kube-rbac-proxy" containerID="cri-o://317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a" gracePeriod=30
May 20 15:48:48.579799 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.579770    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"]
May 20 15:48:48.580045 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580032    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerName="kube-rbac-proxy"
May 20 15:48:48.580087 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580048    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerName="kube-rbac-proxy"
May 20 15:48:48.580087 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580055    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container"
May 20 15:48:48.580087 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580060    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container"
May 20 15:48:48.580087 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580069    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy"
May 20 15:48:48.580087 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580074    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy"
May 20 15:48:48.580087 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580081    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerName="kserve-container"
May 20 15:48:48.580087 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580086    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerName="kserve-container"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580091    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580110    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580128    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="storage-initializer"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580133    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="storage-initializer"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580172    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kube-rbac-proxy"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580181    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerName="kube-rbac-proxy"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580191    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="agent"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580201    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="b5fcc889-3bf1-46bc-bdbd-bdcc73f16461" containerName="kserve-container"
May 20 15:48:48.580325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.580208    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="037d7570-b63a-448b-81f9-52f6bdb24e9a" containerName="kserve-container"
May 20 15:48:48.584196 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.584180    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.586349 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.586331    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
May 20 15:48:48.586606 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.586591    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
May 20 15:48:48.593767 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.593744    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"]
May 20 15:48:48.707687 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.707665    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvhn\" (UniqueName: \"kubernetes.io/projected/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kube-api-access-fpvhn\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.707800 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.707701    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.707800 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.707732    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9e8ca7-f98c-402a-9671-edc1626aee3f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.707908 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.707801    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.808995 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.808929    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9e8ca7-f98c-402a-9671-edc1626aee3f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.808995 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.808980    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.809186 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.809031    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvhn\" (UniqueName: \"kubernetes.io/projected/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kube-api-access-fpvhn\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.809186 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.809072    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.809301 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:48:48.809246    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 20 15:48:48.809353 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:48:48.809330    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls podName:4a9e8ca7-f98c-402a-9671-edc1626aee3f nodeName:}" failed. No retries permitted until 2026-05-20 15:48:49.30930676 +0000 UTC m=+972.000562673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls") pod "isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" (UID: "4a9e8ca7-f98c-402a-9671-edc1626aee3f") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 20 15:48:48.809461 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.809436    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.809668 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.809651    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9e8ca7-f98c-402a-9671-edc1626aee3f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:48.817771 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:48.817748    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvhn\" (UniqueName: \"kubernetes.io/projected/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kube-api-access-fpvhn\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:49.313720 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:49.313688    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:49.316090 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:49.316069    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:49.494892 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:49.494866    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:49.573692 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:49.573657    2556 generic.go:358] "Generic (PLEG): container finished" podID="de721d79-c885-4a21-a554-1863c81cdc52" containerID="317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a" exitCode=2
May 20 15:48:49.573826 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:49.573696    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" event={"ID":"de721d79-c885-4a21-a554-1863c81cdc52","Type":"ContainerDied","Data":"317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a"}
May 20 15:48:49.619233 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:49.619197    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"]
May 20 15:48:49.621679 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:48:49.621652    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9e8ca7_f98c_402a_9671_edc1626aee3f.slice/crio-fb4410bbf407819e6800d8c6a62e49444213b90385364a0c3449815e9797ac30 WatchSource:0}: Error finding container fb4410bbf407819e6800d8c6a62e49444213b90385364a0c3449815e9797ac30: Status 404 returned error can't find the container with id fb4410bbf407819e6800d8c6a62e49444213b90385364a0c3449815e9797ac30
May 20 15:48:50.582458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:50.582421    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" event={"ID":"4a9e8ca7-f98c-402a-9671-edc1626aee3f","Type":"ContainerStarted","Data":"747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9"}
May 20 15:48:50.582458 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:50.582460    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" event={"ID":"4a9e8ca7-f98c-402a-9671-edc1626aee3f","Type":"ContainerStarted","Data":"fb4410bbf407819e6800d8c6a62e49444213b90385364a0c3449815e9797ac30"}
May 20 15:48:52.356971 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.356935    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
May 20 15:48:52.527371 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.527350    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:48:52.589774 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.589706    2556 generic.go:358] "Generic (PLEG): container finished" podID="de721d79-c885-4a21-a554-1863c81cdc52" containerID="8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34" exitCode=0
May 20 15:48:52.589774 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.589756    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" event={"ID":"de721d79-c885-4a21-a554-1863c81cdc52","Type":"ContainerDied","Data":"8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34"}
May 20 15:48:52.589940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.589784    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd" event={"ID":"de721d79-c885-4a21-a554-1863c81cdc52","Type":"ContainerDied","Data":"7546cb8746bed9d7faa2f23672a1878a33f16ec823c53f145bba64fd96f00bfc"}
May 20 15:48:52.589940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.589789    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"
May 20 15:48:52.589940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.589804    2556 scope.go:117] "RemoveContainer" containerID="317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a"
May 20 15:48:52.596623 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.596607    2556 scope.go:117] "RemoveContainer" containerID="8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34"
May 20 15:48:52.602893 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.602877    2556 scope.go:117] "RemoveContainer" containerID="8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69"
May 20 15:48:52.608996 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.608976    2556 scope.go:117] "RemoveContainer" containerID="317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a"
May 20 15:48:52.609229 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:48:52.609212    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a\": container with ID starting with 317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a not found: ID does not exist" containerID="317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a"
May 20 15:48:52.609302 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.609238    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a"} err="failed to get container status \"317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a\": rpc error: code = NotFound desc = could not find container \"317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a\": container with ID starting with 317ab592c46749ed1f0497448efdd7681e59b74f0be692f30d2b64215ad6358a not found: ID does not exist"
May 20 15:48:52.609302 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.609255    2556 scope.go:117] "RemoveContainer" containerID="8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34"
May 20 15:48:52.609489 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:48:52.609475    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34\": container with ID starting with 8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34 not found: ID does not exist" containerID="8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34"
May 20 15:48:52.609534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.609494    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34"} err="failed to get container status \"8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34\": rpc error: code = NotFound desc = could not find container \"8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34\": container with ID starting with 8f2547502a6b785176b25ec18eb4f720cbc69db3f812e996f3490c511c4c7f34 not found: ID does not exist"
May 20 15:48:52.609534 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.609507    2556 scope.go:117] "RemoveContainer" containerID="8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69"
May 20 15:48:52.609735 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:48:52.609715    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69\": container with ID starting with 8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69 not found: ID does not exist" containerID="8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69"
May 20 15:48:52.609781 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.609742    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69"} err="failed to get container status \"8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69\": rpc error: code = NotFound desc = could not find container \"8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69\": container with ID starting with 8733f052b6ab456b8c1e28d244fe7cdf41d4ef5745aa9b40193bf27942821e69 not found: ID does not exist"
May 20 15:48:52.635280 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.635263    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de721d79-c885-4a21-a554-1863c81cdc52-kserve-provision-location\") pod \"de721d79-c885-4a21-a554-1863c81cdc52\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") "
May 20 15:48:52.635352 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.635328    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de721d79-c885-4a21-a554-1863c81cdc52-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"de721d79-c885-4a21-a554-1863c81cdc52\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") "
May 20 15:48:52.635390 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.635368    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkfwv\" (UniqueName: \"kubernetes.io/projected/de721d79-c885-4a21-a554-1863c81cdc52-kube-api-access-kkfwv\") pod \"de721d79-c885-4a21-a554-1863c81cdc52\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") "
May 20 15:48:52.635423 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.635388    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de721d79-c885-4a21-a554-1863c81cdc52-proxy-tls\") pod \"de721d79-c885-4a21-a554-1863c81cdc52\" (UID: \"de721d79-c885-4a21-a554-1863c81cdc52\") "
May 20 15:48:52.635604 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.635579    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de721d79-c885-4a21-a554-1863c81cdc52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de721d79-c885-4a21-a554-1863c81cdc52" (UID: "de721d79-c885-4a21-a554-1863c81cdc52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:48:52.635694 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.635676    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de721d79-c885-4a21-a554-1863c81cdc52-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "de721d79-c885-4a21-a554-1863c81cdc52" (UID: "de721d79-c885-4a21-a554-1863c81cdc52"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:48:52.637269 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.637241    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de721d79-c885-4a21-a554-1863c81cdc52-kube-api-access-kkfwv" (OuterVolumeSpecName: "kube-api-access-kkfwv") pod "de721d79-c885-4a21-a554-1863c81cdc52" (UID: "de721d79-c885-4a21-a554-1863c81cdc52"). InnerVolumeSpecName "kube-api-access-kkfwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:48:52.637349 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.637289    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de721d79-c885-4a21-a554-1863c81cdc52-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de721d79-c885-4a21-a554-1863c81cdc52" (UID: "de721d79-c885-4a21-a554-1863c81cdc52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:48:52.736267 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.736243    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de721d79-c885-4a21-a554-1863c81cdc52-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:48:52.736267 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.736266    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de721d79-c885-4a21-a554-1863c81cdc52-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:48:52.736386 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.736277    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfwv\" (UniqueName: \"kubernetes.io/projected/de721d79-c885-4a21-a554-1863c81cdc52-kube-api-access-kkfwv\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:48:52.736386 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.736286    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de721d79-c885-4a21-a554-1863c81cdc52-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:48:52.913117 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.913073    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"]
May 20 15:48:52.916750 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:52.916728    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-2mztd"]
May 20 15:48:53.594318 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:53.594288    2556 generic.go:358] "Generic (PLEG): container finished" podID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerID="747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9" exitCode=0
May 20 15:48:53.594669 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:53.594347    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" event={"ID":"4a9e8ca7-f98c-402a-9671-edc1626aee3f","Type":"ContainerDied","Data":"747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9"}
May 20 15:48:53.890795 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:53.890727    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de721d79-c885-4a21-a554-1863c81cdc52" path="/var/lib/kubelet/pods/de721d79-c885-4a21-a554-1863c81cdc52/volumes"
May 20 15:48:54.599667 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:54.599627    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" event={"ID":"4a9e8ca7-f98c-402a-9671-edc1626aee3f","Type":"ContainerStarted","Data":"cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24"}
May 20 15:48:54.599667 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:54.599668    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" event={"ID":"4a9e8ca7-f98c-402a-9671-edc1626aee3f","Type":"ContainerStarted","Data":"9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc"}
May 20 15:48:54.600150 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:54.599894    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:54.618992 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:54.618947    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podStartSLOduration=6.618933494 podStartE2EDuration="6.618933494s" podCreationTimestamp="2026-05-20 15:48:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:48:54.618343614 +0000 UTC m=+977.309599566" watchObservedRunningTime="2026-05-20 15:48:54.618933494 +0000 UTC m=+977.310189421"
May 20 15:48:55.603068 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:55.603033    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:48:55.604434 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:55.604392    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:48:56.606470 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:48:56.606383    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:49:01.612540 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:49:01.612502    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:49:01.613238 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:49:01.613205    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:49:11.613993 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:49:11.613954    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:49:21.613486 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:49:21.613451    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:49:31.613230 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:49:31.613183    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:49:41.613896 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:49:41.613845    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:49:51.613243 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:49:51.613203    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:50:01.613030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:01.612996    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:50:11.614278 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:11.614242    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:50:18.892579 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.892546    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"]
May 20 15:50:18.893064 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.892853    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" containerID="cri-o://9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc" gracePeriod=30
May 20 15:50:18.893064 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.892944    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kube-rbac-proxy" containerID="cri-o://cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24" gracePeriod=30
May 20 15:50:18.991512 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991488    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"]
May 20 15:50:18.991751 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991739    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container"
May 20 15:50:18.991800 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991753    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container"
May 20 15:50:18.991800 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991765    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="storage-initializer"
May 20 15:50:18.991800 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991771    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="storage-initializer"
May 20 15:50:18.991800 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991785    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kube-rbac-proxy"
May 20 15:50:18.991800 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991790    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kube-rbac-proxy"
May 20 15:50:18.991960 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991846    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kserve-container"
May 20 15:50:18.991960 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.991860    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="de721d79-c885-4a21-a554-1863c81cdc52" containerName="kube-rbac-proxy"
May 20 15:50:18.994836 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.994821    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:18.997210 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.997187    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
May 20 15:50:18.997324 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:18.997298    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
May 20 15:50:19.006513 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.006494    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"]
May 20 15:50:19.142149 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.142124    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841526a8-4397-480f-999b-80c2c9e15ffe-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.142238 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.142171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841526a8-4397-480f-999b-80c2c9e15ffe-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.142238 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.142199    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xt9f\" (UniqueName: \"kubernetes.io/projected/841526a8-4397-480f-999b-80c2c9e15ffe-kube-api-access-9xt9f\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.142238 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.142223    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.242767 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.242741    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841526a8-4397-480f-999b-80c2c9e15ffe-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.242878 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.242775    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9xt9f\" (UniqueName: \"kubernetes.io/projected/841526a8-4397-480f-999b-80c2c9e15ffe-kube-api-access-9xt9f\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.242878 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.242799    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.242878 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.242843    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841526a8-4397-480f-999b-80c2c9e15ffe-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.243042 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:50:19.243003    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
May 20 15:50:19.243120 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:50:19.243071    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls podName:841526a8-4397-480f-999b-80c2c9e15ffe nodeName:}" failed. No retries permitted until 2026-05-20 15:50:19.743050501 +0000 UTC m=+1062.434306410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" (UID: "841526a8-4397-480f-999b-80c2c9e15ffe") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
May 20 15:50:19.243198 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.243179    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841526a8-4397-480f-999b-80c2c9e15ffe-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.243435 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.243416    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841526a8-4397-480f-999b-80c2c9e15ffe-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.254860 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.254838    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xt9f\" (UniqueName: \"kubernetes.io/projected/841526a8-4397-480f-999b-80c2c9e15ffe-kube-api-access-9xt9f\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.746822 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.746795    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.749046 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.749029    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:19.842593 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.842553    2556 generic.go:358] "Generic (PLEG): container finished" podID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerID="cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24" exitCode=2
May 20 15:50:19.842745 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.842636    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" event={"ID":"4a9e8ca7-f98c-402a-9671-edc1626aee3f","Type":"ContainerDied","Data":"cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24"}
May 20 15:50:19.904785 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:19.904758    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:50:20.038163 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:20.038077    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"]
May 20 15:50:20.041062 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:50:20.041036    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841526a8_4397_480f_999b_80c2c9e15ffe.slice/crio-53e03af929164373c17ef4ed31e8d82daaaaab5b0dfdb05b0c815d6f00486aad WatchSource:0}: Error finding container 53e03af929164373c17ef4ed31e8d82daaaaab5b0dfdb05b0c815d6f00486aad: Status 404 returned error can't find the container with id 53e03af929164373c17ef4ed31e8d82daaaaab5b0dfdb05b0c815d6f00486aad
May 20 15:50:20.846816 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:20.846774    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" event={"ID":"841526a8-4397-480f-999b-80c2c9e15ffe","Type":"ContainerStarted","Data":"9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302"}
May 20 15:50:20.846985 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:20.846823    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" event={"ID":"841526a8-4397-480f-999b-80c2c9e15ffe","Type":"ContainerStarted","Data":"53e03af929164373c17ef4ed31e8d82daaaaab5b0dfdb05b0c815d6f00486aad"}
May 20 15:50:21.606877 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:21.606834    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
May 20 15:50:21.613052 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:21.613022    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
May 20 15:50:23.032739 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.032718    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:50:23.172587 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.172509    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpvhn\" (UniqueName: \"kubernetes.io/projected/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kube-api-access-fpvhn\") pod \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") "
May 20 15:50:23.172587 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.172548    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls\") pod \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") "
May 20 15:50:23.172587 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.172581    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kserve-provision-location\") pod \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") "
May 20 15:50:23.172856 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.172622    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9e8ca7-f98c-402a-9671-edc1626aee3f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\" (UID: \"4a9e8ca7-f98c-402a-9671-edc1626aee3f\") "
May 20 15:50:23.172957 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.172932    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a9e8ca7-f98c-402a-9671-edc1626aee3f" (UID: "4a9e8ca7-f98c-402a-9671-edc1626aee3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:50:23.173000 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.172977    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9e8ca7-f98c-402a-9671-edc1626aee3f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "4a9e8ca7-f98c-402a-9671-edc1626aee3f" (UID: "4a9e8ca7-f98c-402a-9671-edc1626aee3f"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:50:23.174591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.174572    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a9e8ca7-f98c-402a-9671-edc1626aee3f" (UID: "4a9e8ca7-f98c-402a-9671-edc1626aee3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:50:23.174684 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.174665    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kube-api-access-fpvhn" (OuterVolumeSpecName: "kube-api-access-fpvhn") pod "4a9e8ca7-f98c-402a-9671-edc1626aee3f" (UID: "4a9e8ca7-f98c-402a-9671-edc1626aee3f"). InnerVolumeSpecName "kube-api-access-fpvhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:50:23.273891 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.273862    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9e8ca7-f98c-402a-9671-edc1626aee3f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:50:23.273891 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.273888    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvhn\" (UniqueName: \"kubernetes.io/projected/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kube-api-access-fpvhn\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:50:23.274043 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.273898    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9e8ca7-f98c-402a-9671-edc1626aee3f-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:50:23.274043 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.273907    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9e8ca7-f98c-402a-9671-edc1626aee3f-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:50:23.855868 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.855836    2556 generic.go:358] "Generic (PLEG): container finished" podID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerID="9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc" exitCode=0
May 20 15:50:23.856034 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.855915    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"
May 20 15:50:23.856034 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.855915    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" event={"ID":"4a9e8ca7-f98c-402a-9671-edc1626aee3f","Type":"ContainerDied","Data":"9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc"}
May 20 15:50:23.856034 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.855956    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh" event={"ID":"4a9e8ca7-f98c-402a-9671-edc1626aee3f","Type":"ContainerDied","Data":"fb4410bbf407819e6800d8c6a62e49444213b90385364a0c3449815e9797ac30"}
May 20 15:50:23.856034 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.855972    2556 scope.go:117] "RemoveContainer" containerID="cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24"
May 20 15:50:23.863728 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.863711    2556 scope.go:117] "RemoveContainer" containerID="9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc"
May 20 15:50:23.870607 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.870595    2556 scope.go:117] "RemoveContainer" containerID="747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9"
May 20 15:50:23.877195 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.877171    2556 scope.go:117] "RemoveContainer" containerID="cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24"
May 20 15:50:23.877505 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:50:23.877480    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24\": container with ID starting with cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24 not found: ID does not exist" containerID="cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24"
May 20 15:50:23.877611 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.877515    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24"} err="failed to get container status \"cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24\": rpc error: code = NotFound desc = could not find container \"cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24\": container with ID starting with cda6f094ddbbfc1872d18e442e7710ab2256caff81ca47b857a831db6f17fa24 not found: ID does not exist"
May 20 15:50:23.877611 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.877543    2556 scope.go:117] "RemoveContainer" containerID="9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc"
May 20 15:50:23.877837 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:50:23.877818    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc\": container with ID starting with 9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc not found: ID does not exist" containerID="9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc"
May 20 15:50:23.877912 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.877844    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc"} err="failed to get container status \"9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc\": rpc error: code = NotFound desc = could not find container \"9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc\": container with ID starting with 9587648dfe081a6c3664de4900dd56cdc6471b2ab4c4cb91851d56f40d40ddbc not found: ID does not exist"
May 20 15:50:23.877912 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.877865    2556 scope.go:117] "RemoveContainer" containerID="747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9"
May 20 15:50:23.877912 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.877882    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"]
May 20 15:50:23.878207 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:50:23.878181    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9\": container with ID starting with 747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9 not found: ID does not exist" containerID="747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9"
May 20 15:50:23.878274 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.878216    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9"} err="failed to get container status \"747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9\": rpc error: code = NotFound desc = could not find container \"747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9\": container with ID starting with 747dfdd38b8962d45cb2d2652c427011fa9b1ac8d7492fe84f8df43fdb211fa9 not found: ID does not exist"
May 20 15:50:23.879995 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.879972    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-k9nhh"]
May 20 15:50:23.890841 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:23.890813    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" path="/var/lib/kubelet/pods/4a9e8ca7-f98c-402a-9671-edc1626aee3f/volumes"
May 20 15:50:24.860400 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:24.860368    2556 generic.go:358] "Generic (PLEG): container finished" podID="841526a8-4397-480f-999b-80c2c9e15ffe" containerID="9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302" exitCode=0
May 20 15:50:24.860903 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:50:24.860434    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" event={"ID":"841526a8-4397-480f-999b-80c2c9e15ffe","Type":"ContainerDied","Data":"9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302"}
May 20 15:52:34.708988 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:52:34.708969    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:52:35.237109 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:52:35.237071    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" event={"ID":"841526a8-4397-480f-999b-80c2c9e15ffe","Type":"ContainerStarted","Data":"381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4"}
May 20 15:52:35.237351 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:52:35.237127    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" event={"ID":"841526a8-4397-480f-999b-80c2c9e15ffe","Type":"ContainerStarted","Data":"d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc"}
May 20 15:52:35.237351 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:52:35.237297    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:52:35.268769 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:52:35.268724    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" podStartSLOduration=7.521799374 podStartE2EDuration="2m17.268711856s" podCreationTimestamp="2026-05-20 15:50:18 +0000 UTC" firstStartedPulling="2026-05-20 15:50:24.861530392 +0000 UTC m=+1067.552786298" lastFinishedPulling="2026-05-20 15:52:34.608442869 +0000 UTC m=+1197.299698780" observedRunningTime="2026-05-20 15:52:35.266611499 +0000 UTC m=+1197.957867426" watchObservedRunningTime="2026-05-20 15:52:35.268711856 +0000 UTC m=+1197.959967783"
May 20 15:52:36.246072 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:52:36.246032    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:52:42.254214 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:52:42.254187    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:53:12.257795 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:12.257762    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:53:19.171667 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.171630    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"]
May 20 15:53:19.173029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.172034    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="kserve-container" containerID="cri-o://d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc" gracePeriod=30
May 20 15:53:19.173029 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.172075    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="kube-rbac-proxy" containerID="cri-o://381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4" gracePeriod=30
May 20 15:53:19.279964 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.279934    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"]
May 20 15:53:19.280232 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.280216    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="storage-initializer"
May 20 15:53:19.280232 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.280233    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="storage-initializer"
May 20 15:53:19.280322 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.280247    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container"
May 20 15:53:19.280322 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.280253    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container"
May 20 15:53:19.280322 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.280265    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kube-rbac-proxy"
May 20 15:53:19.280322 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.280272    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kube-rbac-proxy"
May 20 15:53:19.280322 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.280314    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kserve-container"
May 20 15:53:19.280322 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.280323    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="4a9e8ca7-f98c-402a-9671-edc1626aee3f" containerName="kube-rbac-proxy"
May 20 15:53:19.282195 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.282174    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.284872 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.284845    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
May 20 15:53:19.284872 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.284861    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
May 20 15:53:19.293387 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.293366    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"]
May 20 15:53:19.337199 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.337169    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.337379 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.337218    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.337379 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.337248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.337379 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.337322    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnmzr\" (UniqueName: \"kubernetes.io/projected/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kube-api-access-bnmzr\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.362039 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.362011    2556 generic.go:358] "Generic (PLEG): container finished" podID="841526a8-4397-480f-999b-80c2c9e15ffe" containerID="381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4" exitCode=2
May 20 15:53:19.362208 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.362071    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" event={"ID":"841526a8-4397-480f-999b-80c2c9e15ffe","Type":"ContainerDied","Data":"381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4"}
May 20 15:53:19.438654 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.438571    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.438654 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.438627    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.438874 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.438665    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.438874 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.438697    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bnmzr\" (UniqueName: \"kubernetes.io/projected/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kube-api-access-bnmzr\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.438874 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:19.438745    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
May 20 15:53:19.438874 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:19.438821    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls podName:f54fb499-dc7d-4865-85c2-68c81d7d7d1c nodeName:}" failed. No retries permitted until 2026-05-20 15:53:19.93879887 +0000 UTC m=+1242.630054781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" (UID: "f54fb499-dc7d-4865-85c2-68c81d7d7d1c") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
May 20 15:53:19.439161 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.439081    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.439382 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.439361    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.447988 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.447958    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnmzr\" (UniqueName: \"kubernetes.io/projected/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kube-api-access-bnmzr\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.943567 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.943538    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:19.945749 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:19.945731    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:20.104505 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.104484    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:53:20.145341 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.145320    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xt9f\" (UniqueName: \"kubernetes.io/projected/841526a8-4397-480f-999b-80c2c9e15ffe-kube-api-access-9xt9f\") pod \"841526a8-4397-480f-999b-80c2c9e15ffe\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") "
May 20 15:53:20.145481 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.145375    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841526a8-4397-480f-999b-80c2c9e15ffe-kserve-provision-location\") pod \"841526a8-4397-480f-999b-80c2c9e15ffe\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") "
May 20 15:53:20.145481 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.145400    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls\") pod \"841526a8-4397-480f-999b-80c2c9e15ffe\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") "
May 20 15:53:20.145598 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.145495    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841526a8-4397-480f-999b-80c2c9e15ffe-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"841526a8-4397-480f-999b-80c2c9e15ffe\" (UID: \"841526a8-4397-480f-999b-80c2c9e15ffe\") "
May 20 15:53:20.145672 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.145652    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841526a8-4397-480f-999b-80c2c9e15ffe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "841526a8-4397-480f-999b-80c2c9e15ffe" (UID: "841526a8-4397-480f-999b-80c2c9e15ffe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:53:20.145752 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.145735    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841526a8-4397-480f-999b-80c2c9e15ffe-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:53:20.145859 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.145819    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841526a8-4397-480f-999b-80c2c9e15ffe-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "841526a8-4397-480f-999b-80c2c9e15ffe" (UID: "841526a8-4397-480f-999b-80c2c9e15ffe"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:53:20.147377 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.147358    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841526a8-4397-480f-999b-80c2c9e15ffe-kube-api-access-9xt9f" (OuterVolumeSpecName: "kube-api-access-9xt9f") pod "841526a8-4397-480f-999b-80c2c9e15ffe" (UID: "841526a8-4397-480f-999b-80c2c9e15ffe"). InnerVolumeSpecName "kube-api-access-9xt9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:53:20.147432 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.147386    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "841526a8-4397-480f-999b-80c2c9e15ffe" (UID: "841526a8-4397-480f-999b-80c2c9e15ffe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:53:20.193690 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.193667    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:20.246740 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.246716    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841526a8-4397-480f-999b-80c2c9e15ffe-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:53:20.246740 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.246739    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841526a8-4397-480f-999b-80c2c9e15ffe-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:53:20.246942 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.246750    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xt9f\" (UniqueName: \"kubernetes.io/projected/841526a8-4397-480f-999b-80c2c9e15ffe-kube-api-access-9xt9f\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:53:20.313208 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.313182    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"]
May 20 15:53:20.315847 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:53:20.315821    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54fb499_dc7d_4865_85c2_68c81d7d7d1c.slice/crio-fa3395bfbc9c59d12d3f0b1445727602b3593db98fe74a3cdc3c433da37a9a2b WatchSource:0}: Error finding container fa3395bfbc9c59d12d3f0b1445727602b3593db98fe74a3cdc3c433da37a9a2b: Status 404 returned error can't find the container with id fa3395bfbc9c59d12d3f0b1445727602b3593db98fe74a3cdc3c433da37a9a2b
May 20 15:53:20.366320 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.366296    2556 generic.go:358] "Generic (PLEG): container finished" podID="841526a8-4397-480f-999b-80c2c9e15ffe" containerID="d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc" exitCode=0
May 20 15:53:20.366420 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.366369    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"
May 20 15:53:20.366482 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.366368    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" event={"ID":"841526a8-4397-480f-999b-80c2c9e15ffe","Type":"ContainerDied","Data":"d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc"}
May 20 15:53:20.366536 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.366482    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs" event={"ID":"841526a8-4397-480f-999b-80c2c9e15ffe","Type":"ContainerDied","Data":"53e03af929164373c17ef4ed31e8d82daaaaab5b0dfdb05b0c815d6f00486aad"}
May 20 15:53:20.366536 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.366503    2556 scope.go:117] "RemoveContainer" containerID="381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4"
May 20 15:53:20.367868 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.367819    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" event={"ID":"f54fb499-dc7d-4865-85c2-68c81d7d7d1c","Type":"ContainerStarted","Data":"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255"}
May 20 15:53:20.367868 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.367849    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" event={"ID":"f54fb499-dc7d-4865-85c2-68c81d7d7d1c","Type":"ContainerStarted","Data":"fa3395bfbc9c59d12d3f0b1445727602b3593db98fe74a3cdc3c433da37a9a2b"}
May 20 15:53:20.379988 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.379970    2556 scope.go:117] "RemoveContainer" containerID="d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc"
May 20 15:53:20.387996 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.387977    2556 scope.go:117] "RemoveContainer" containerID="9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302"
May 20 15:53:20.395336 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.395308    2556 scope.go:117] "RemoveContainer" containerID="381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4"
May 20 15:53:20.395598 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:20.395578    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4\": container with ID starting with 381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4 not found: ID does not exist" containerID="381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4"
May 20 15:53:20.395655 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.395605    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4"} err="failed to get container status \"381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4\": rpc error: code = NotFound desc = could not find container \"381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4\": container with ID starting with 381027fe3ad64d8c305383fb45975feac84fc1a05e1d6df12a343c021ecaf4d4 not found: ID does not exist"
May 20 15:53:20.395655 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.395623    2556 scope.go:117] "RemoveContainer" containerID="d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc"
May 20 15:53:20.395876 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:20.395857    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc\": container with ID starting with d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc not found: ID does not exist" containerID="d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc"
May 20 15:53:20.395926 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.395880    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc"} err="failed to get container status \"d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc\": rpc error: code = NotFound desc = could not find container \"d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc\": container with ID starting with d8ab11a8287cd16f7ab79b374a8d4151229a6a783c0683af2fdb967203e970cc not found: ID does not exist"
May 20 15:53:20.395926 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.395893    2556 scope.go:117] "RemoveContainer" containerID="9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302"
May 20 15:53:20.396077 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:20.396062    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302\": container with ID starting with 9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302 not found: ID does not exist" containerID="9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302"
May 20 15:53:20.396138 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.396083    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302"} err="failed to get container status \"9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302\": rpc error: code = NotFound desc = could not find container \"9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302\": container with ID starting with 9ead867ad088fb1919fbb4b81e5090e49c7d03bc586eed58ea486488346b9302 not found: ID does not exist"
May 20 15:53:20.404511 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.404493    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"]
May 20 15:53:20.408959 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:20.408942    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-xk9cs"]
May 20 15:53:21.890699 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:21.890660    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" path="/var/lib/kubelet/pods/841526a8-4397-480f-999b-80c2c9e15ffe/volumes"
May 20 15:53:24.382371 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:24.382300    2556 generic.go:358] "Generic (PLEG): container finished" podID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerID="7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255" exitCode=0
May 20 15:53:24.382666 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:24.382377    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" event={"ID":"f54fb499-dc7d-4865-85c2-68c81d7d7d1c","Type":"ContainerDied","Data":"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255"}
May 20 15:53:25.386956 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:25.386924    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" event={"ID":"f54fb499-dc7d-4865-85c2-68c81d7d7d1c","Type":"ContainerStarted","Data":"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b"}
May 20 15:53:25.387342 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:25.386964    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" event={"ID":"f54fb499-dc7d-4865-85c2-68c81d7d7d1c","Type":"ContainerStarted","Data":"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30"}
May 20 15:53:25.387342 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:25.387209    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:25.406053 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:25.405997    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" podStartSLOduration=6.405980414 podStartE2EDuration="6.405980414s" podCreationTimestamp="2026-05-20 15:53:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:53:25.404758917 +0000 UTC m=+1248.096014854" watchObservedRunningTime="2026-05-20 15:53:25.405980414 +0000 UTC m=+1248.097236342"
May 20 15:53:26.390076 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:26.390003    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:26.391374 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:26.391346    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 20 15:53:27.392727 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:27.392686    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
May 20 15:53:32.397843 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:32.397808    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:32.398999 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:32.398976    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:39.319991 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.319954    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"]
May 20 15:53:39.320478 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.320425    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kserve-container" containerID="cri-o://7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30" gracePeriod=30
May 20 15:53:39.320604 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.320517    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kube-rbac-proxy" containerID="cri-o://bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b" gracePeriod=30
May 20 15:53:39.397503 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397447    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"]
May 20 15:53:39.397716 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397703    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="storage-initializer"
May 20 15:53:39.397757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397718    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="storage-initializer"
May 20 15:53:39.397757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397729    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="kube-rbac-proxy"
May 20 15:53:39.397757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397735    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="kube-rbac-proxy"
May 20 15:53:39.397757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397744    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="kserve-container"
May 20 15:53:39.397757 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397750    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="kserve-container"
May 20 15:53:39.397906 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397789    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="kube-rbac-proxy"
May 20 15:53:39.397906 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.397823    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="841526a8-4397-480f-999b-80c2c9e15ffe" containerName="kserve-container"
May 20 15:53:39.400320 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.400306    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.402718 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.402702    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
May 20 15:53:39.402909 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.402895    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
May 20 15:53:39.410272 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.410250    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"]
May 20 15:53:39.480420 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.480401    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjx9\" (UniqueName: \"kubernetes.io/projected/bc0c5586-15de-4047-bf24-1afff14a6de4-kube-api-access-rkjx9\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.480507 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.480432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.480507 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.480459    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc0c5586-15de-4047-bf24-1afff14a6de4-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.480596 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.480565    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc0c5586-15de-4047-bf24-1afff14a6de4-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.581783 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.581738    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjx9\" (UniqueName: \"kubernetes.io/projected/bc0c5586-15de-4047-bf24-1afff14a6de4-kube-api-access-rkjx9\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.581959 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.581804    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.581959 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.581848    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc0c5586-15de-4047-bf24-1afff14a6de4-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.581959 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.581914    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc0c5586-15de-4047-bf24-1afff14a6de4-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.582151 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:39.581952    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
May 20 15:53:39.582151 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:39.582048    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls podName:bc0c5586-15de-4047-bf24-1afff14a6de4 nodeName:}" failed. No retries permitted until 2026-05-20 15:53:40.082030635 +0000 UTC m=+1262.773286549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" (UID: "bc0c5586-15de-4047-bf24-1afff14a6de4") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
May 20 15:53:39.582438 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.582412    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc0c5586-15de-4047-bf24-1afff14a6de4-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.582656 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.582635    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc0c5586-15de-4047-bf24-1afff14a6de4-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.591482 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.591462    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjx9\" (UniqueName: \"kubernetes.io/projected/bc0c5586-15de-4047-bf24-1afff14a6de4-kube-api-access-rkjx9\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:39.962790 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:39.962769    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:40.085750 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.085696    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") "
May 20 15:53:40.085750 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.085726    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kserve-provision-location\") pod \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") "
May 20 15:53:40.085750 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.085747    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls\") pod \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") "
May 20 15:53:40.085960 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.085803    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnmzr\" (UniqueName: \"kubernetes.io/projected/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kube-api-access-bnmzr\") pod \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\" (UID: \"f54fb499-dc7d-4865-85c2-68c81d7d7d1c\") "
May 20 15:53:40.085960 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.085922    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:40.086072 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.086002    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "f54fb499-dc7d-4865-85c2-68c81d7d7d1c" (UID: "f54fb499-dc7d-4865-85c2-68c81d7d7d1c"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:53:40.086072 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.086005    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f54fb499-dc7d-4865-85c2-68c81d7d7d1c" (UID: "f54fb499-dc7d-4865-85c2-68c81d7d7d1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:53:40.087862 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.087839    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f54fb499-dc7d-4865-85c2-68c81d7d7d1c" (UID: "f54fb499-dc7d-4865-85c2-68c81d7d7d1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:53:40.087951 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.087891    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kube-api-access-bnmzr" (OuterVolumeSpecName: "kube-api-access-bnmzr") pod "f54fb499-dc7d-4865-85c2-68c81d7d7d1c" (UID: "f54fb499-dc7d-4865-85c2-68c81d7d7d1c"). InnerVolumeSpecName "kube-api-access-bnmzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:53:40.088198 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.088182    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:40.186384 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.186364    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnmzr\" (UniqueName: \"kubernetes.io/projected/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kube-api-access-bnmzr\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:53:40.186384 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.186384    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:53:40.186493 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.186395    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:53:40.186493 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.186405    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f54fb499-dc7d-4865-85c2-68c81d7d7d1c-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:53:40.309898 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.309873    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:40.428608 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.428575    2556 generic.go:358] "Generic (PLEG): container finished" podID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerID="bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b" exitCode=2
May 20 15:53:40.428608 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.428607    2556 generic.go:358] "Generic (PLEG): container finished" podID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerID="7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30" exitCode=0
May 20 15:53:40.429067 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.428657    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" event={"ID":"f54fb499-dc7d-4865-85c2-68c81d7d7d1c","Type":"ContainerDied","Data":"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b"}
May 20 15:53:40.429067 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.428696    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"
May 20 15:53:40.429067 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.428718    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" event={"ID":"f54fb499-dc7d-4865-85c2-68c81d7d7d1c","Type":"ContainerDied","Data":"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30"}
May 20 15:53:40.429067 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.428730    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq" event={"ID":"f54fb499-dc7d-4865-85c2-68c81d7d7d1c","Type":"ContainerDied","Data":"fa3395bfbc9c59d12d3f0b1445727602b3593db98fe74a3cdc3c433da37a9a2b"}
May 20 15:53:40.429067 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.428745    2556 scope.go:117] "RemoveContainer" containerID="bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b"
May 20 15:53:40.438162 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.438140    2556 scope.go:117] "RemoveContainer" containerID="7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30"
May 20 15:53:40.443477 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.443459    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"]
May 20 15:53:40.445381 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.445360    2556 scope.go:117] "RemoveContainer" containerID="7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255"
May 20 15:53:40.447170 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:53:40.447146    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0c5586_15de_4047_bf24_1afff14a6de4.slice/crio-4931c43da23c75506c7ded30ad6708919c9bd757cca1fcb785a7791ebeec23b9 WatchSource:0}: Error finding container 4931c43da23c75506c7ded30ad6708919c9bd757cca1fcb785a7791ebeec23b9: Status 404 returned error can't find the container with id 4931c43da23c75506c7ded30ad6708919c9bd757cca1fcb785a7791ebeec23b9
May 20 15:53:40.452894 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.452873    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"]
May 20 15:53:40.456352 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.456331    2556 scope.go:117] "RemoveContainer" containerID="bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b"
May 20 15:53:40.456714 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:40.456684    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b\": container with ID starting with bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b not found: ID does not exist" containerID="bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b"
May 20 15:53:40.456790 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.456726    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b"} err="failed to get container status \"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b\": rpc error: code = NotFound desc = could not find container \"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b\": container with ID starting with bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b not found: ID does not exist"
May 20 15:53:40.456790 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.456751    2556 scope.go:117] "RemoveContainer" containerID="7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30"
May 20 15:53:40.457069 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:40.457049    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30\": container with ID starting with 7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30 not found: ID does not exist" containerID="7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30"
May 20 15:53:40.457161 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.457077    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30"} err="failed to get container status \"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30\": rpc error: code = NotFound desc = could not find container \"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30\": container with ID starting with 7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30 not found: ID does not exist"
May 20 15:53:40.457161 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.457120    2556 scope.go:117] "RemoveContainer" containerID="7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255"
May 20 15:53:40.457482 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:53:40.457462    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255\": container with ID starting with 7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255 not found: ID does not exist" containerID="7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255"
May 20 15:53:40.457604 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.457483    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255"} err="failed to get container status \"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255\": rpc error: code = NotFound desc = could not find container \"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255\": container with ID starting with 7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255 not found: ID does not exist"
May 20 15:53:40.457604 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.457498    2556 scope.go:117] "RemoveContainer" containerID="bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b"
May 20 15:53:40.457726 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.457689    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8lrhq"]
May 20 15:53:40.457782 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.457735    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b"} err="failed to get container status \"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b\": rpc error: code = NotFound desc = could not find container \"bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b\": container with ID starting with bab82f84ca3beb2c1ac19fcbe5d4391d067cd1bb28ae30f0847a114a3ee4542b not found: ID does not exist"
May 20 15:53:40.457782 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.457758    2556 scope.go:117] "RemoveContainer" containerID="7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30"
May 20 15:53:40.458007 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.457981    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30"} err="failed to get container status \"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30\": rpc error: code = NotFound desc = could not find container \"7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30\": container with ID starting with 7545c5101c03f9f342d1d36906da7d1a1f3564589e214bd003a6fce782bbcb30 not found: ID does not exist"
May 20 15:53:40.458071 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.458010    2556 scope.go:117] "RemoveContainer" containerID="7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255"
May 20 15:53:40.458252 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:40.458232    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255"} err="failed to get container status \"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255\": rpc error: code = NotFound desc = could not find container \"7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255\": container with ID starting with 7030933c39b17af87ea036f7cb72f19434aa674b15ec5b0df84ee9ae1ea51255 not found: ID does not exist"
May 20 15:53:41.433705 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:41.433659    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" event={"ID":"bc0c5586-15de-4047-bf24-1afff14a6de4","Type":"ContainerStarted","Data":"00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc"}
May 20 15:53:41.433705 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:41.433709    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" event={"ID":"bc0c5586-15de-4047-bf24-1afff14a6de4","Type":"ContainerStarted","Data":"4931c43da23c75506c7ded30ad6708919c9bd757cca1fcb785a7791ebeec23b9"}
May 20 15:53:41.891316 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:41.891279    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" path="/var/lib/kubelet/pods/f54fb499-dc7d-4865-85c2-68c81d7d7d1c/volumes"
May 20 15:53:44.443621 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:44.443592    2556 generic.go:358] "Generic (PLEG): container finished" podID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerID="00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc" exitCode=0
May 20 15:53:44.444005 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:44.443666    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" event={"ID":"bc0c5586-15de-4047-bf24-1afff14a6de4","Type":"ContainerDied","Data":"00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc"}
May 20 15:53:45.448464 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:45.448430    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" event={"ID":"bc0c5586-15de-4047-bf24-1afff14a6de4","Type":"ContainerStarted","Data":"c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4"}
May 20 15:53:45.448464 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:45.448467    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" event={"ID":"bc0c5586-15de-4047-bf24-1afff14a6de4","Type":"ContainerStarted","Data":"a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a"}
May 20 15:53:45.448920 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:45.448665    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:45.448920 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:45.448687    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:53:45.474008 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:45.469673    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" podStartSLOduration=6.469659073 podStartE2EDuration="6.469659073s" podCreationTimestamp="2026-05-20 15:53:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:53:45.467450157 +0000 UTC m=+1268.158706086" watchObservedRunningTime="2026-05-20 15:53:45.469659073 +0000 UTC m=+1268.160915002"
May 20 15:53:51.460207 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:53:51.460172    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:54:21.464092 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:21.464065    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:54:29.608934 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.608903    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"]
May 20 15:54:29.609331 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.609258    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="kserve-container" containerID="cri-o://a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a" gracePeriod=30
May 20 15:54:29.609403 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.609300    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="kube-rbac-proxy" containerID="cri-o://c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4" gracePeriod=30
May 20 15:54:29.682905 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.682872    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"]
May 20 15:54:29.683174 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.683162    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kube-rbac-proxy"
May 20 15:54:29.683228 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.683175    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kube-rbac-proxy"
May 20 15:54:29.683228 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.683188    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="storage-initializer"
May 20 15:54:29.683228 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.683194    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="storage-initializer"
May 20 15:54:29.683228 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.683202    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kserve-container"
May 20 15:54:29.683228 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.683208    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kserve-container"
May 20 15:54:29.683379 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.683253    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kserve-container"
May 20 15:54:29.683379 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.683260    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="f54fb499-dc7d-4865-85c2-68c81d7d7d1c" containerName="kube-rbac-proxy"
May 20 15:54:29.685155 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.685140    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.687744 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.687720    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
May 20 15:54:29.687861 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.687751    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
May 20 15:54:29.698535 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.698511    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"]
May 20 15:54:29.830457 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.830422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.830457 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.830458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4384a77-580e-43b8-868b-6244e69994cc-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.830706 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.830497    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlg2\" (UniqueName: \"kubernetes.io/projected/f4384a77-580e-43b8-868b-6244e69994cc-kube-api-access-8zlg2\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.830706 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.830609    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4384a77-580e-43b8-868b-6244e69994cc-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.931572 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.931478    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4384a77-580e-43b8-868b-6244e69994cc-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.931572 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.931561    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.931782 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.931586    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4384a77-580e-43b8-868b-6244e69994cc-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.931782 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.931617    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlg2\" (UniqueName: \"kubernetes.io/projected/f4384a77-580e-43b8-868b-6244e69994cc-kube-api-access-8zlg2\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.931782 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:54:29.931727    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 20 15:54:29.931918 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:54:29.931798    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls podName:f4384a77-580e-43b8-868b-6244e69994cc nodeName:}" failed. No retries permitted until 2026-05-20 15:54:30.431779168 +0000 UTC m=+1313.123035081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls") pod "isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" (UID: "f4384a77-580e-43b8-868b-6244e69994cc") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 20 15:54:29.931918 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.931903    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4384a77-580e-43b8-868b-6244e69994cc-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.932225 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.932209    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4384a77-580e-43b8-868b-6244e69994cc-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:29.941021 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:29.940994    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlg2\" (UniqueName: \"kubernetes.io/projected/f4384a77-580e-43b8-868b-6244e69994cc-kube-api-access-8zlg2\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:30.435965 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.435935    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:30.438320 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.438296    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:30.578137 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.578093    2556 generic.go:358] "Generic (PLEG): container finished" podID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerID="c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4" exitCode=2
May 20 15:54:30.578247 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.578169    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" event={"ID":"bc0c5586-15de-4047-bf24-1afff14a6de4","Type":"ContainerDied","Data":"c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4"}
May 20 15:54:30.594337 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.594319    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:30.776809 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.776790    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:54:30.939340 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.939295    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"]
May 20 15:54:30.940131 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.940113    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls\") pod \"bc0c5586-15de-4047-bf24-1afff14a6de4\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") "
May 20 15:54:30.940190 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.940153    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc0c5586-15de-4047-bf24-1afff14a6de4-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"bc0c5586-15de-4047-bf24-1afff14a6de4\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") "
May 20 15:54:30.940249 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.940207    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc0c5586-15de-4047-bf24-1afff14a6de4-kserve-provision-location\") pod \"bc0c5586-15de-4047-bf24-1afff14a6de4\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") "
May 20 15:54:30.940301 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.940248    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkjx9\" (UniqueName: \"kubernetes.io/projected/bc0c5586-15de-4047-bf24-1afff14a6de4-kube-api-access-rkjx9\") pod \"bc0c5586-15de-4047-bf24-1afff14a6de4\" (UID: \"bc0c5586-15de-4047-bf24-1afff14a6de4\") "
May 20 15:54:30.940564 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.940539    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0c5586-15de-4047-bf24-1afff14a6de4-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "bc0c5586-15de-4047-bf24-1afff14a6de4" (UID: "bc0c5586-15de-4047-bf24-1afff14a6de4"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:54:30.940665 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.940596    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0c5586-15de-4047-bf24-1afff14a6de4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc0c5586-15de-4047-bf24-1afff14a6de4" (UID: "bc0c5586-15de-4047-bf24-1afff14a6de4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:54:30.942252 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.942227    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc0c5586-15de-4047-bf24-1afff14a6de4" (UID: "bc0c5586-15de-4047-bf24-1afff14a6de4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:54:30.942347 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:54:30.942264    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4384a77_580e_43b8_868b_6244e69994cc.slice/crio-7f8450f99821dad2c0c1a9e43cd399abb74cd477052e60a34a62f95aae176667 WatchSource:0}: Error finding container 7f8450f99821dad2c0c1a9e43cd399abb74cd477052e60a34a62f95aae176667: Status 404 returned error can't find the container with id 7f8450f99821dad2c0c1a9e43cd399abb74cd477052e60a34a62f95aae176667
May 20 15:54:30.942412 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:30.942395    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0c5586-15de-4047-bf24-1afff14a6de4-kube-api-access-rkjx9" (OuterVolumeSpecName: "kube-api-access-rkjx9") pod "bc0c5586-15de-4047-bf24-1afff14a6de4" (UID: "bc0c5586-15de-4047-bf24-1afff14a6de4"). InnerVolumeSpecName "kube-api-access-rkjx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:54:31.040982 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.040957    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc0c5586-15de-4047-bf24-1afff14a6de4-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:54:31.040982 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.040979    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkjx9\" (UniqueName: \"kubernetes.io/projected/bc0c5586-15de-4047-bf24-1afff14a6de4-kube-api-access-rkjx9\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:54:31.041124 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.040989    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc0c5586-15de-4047-bf24-1afff14a6de4-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:54:31.041124 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.040999    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc0c5586-15de-4047-bf24-1afff14a6de4-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:54:31.582291 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.582255    2556 generic.go:358] "Generic (PLEG): container finished" podID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerID="a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a" exitCode=0
May 20 15:54:31.582475 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.582340    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"
May 20 15:54:31.582475 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.582341    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" event={"ID":"bc0c5586-15de-4047-bf24-1afff14a6de4","Type":"ContainerDied","Data":"a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a"}
May 20 15:54:31.582475 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.582378    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g" event={"ID":"bc0c5586-15de-4047-bf24-1afff14a6de4","Type":"ContainerDied","Data":"4931c43da23c75506c7ded30ad6708919c9bd757cca1fcb785a7791ebeec23b9"}
May 20 15:54:31.582475 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.582400    2556 scope.go:117] "RemoveContainer" containerID="c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4"
May 20 15:54:31.583767 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.583746    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerStarted","Data":"a983854a356e0811ffc48c2e539ef879f46806a0837b9b8415a34f0d9487eb49"}
May 20 15:54:31.583856 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.583777    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerStarted","Data":"7f8450f99821dad2c0c1a9e43cd399abb74cd477052e60a34a62f95aae176667"}
May 20 15:54:31.590026 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.590006    2556 scope.go:117] "RemoveContainer" containerID="a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a"
May 20 15:54:31.596837 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.596821    2556 scope.go:117] "RemoveContainer" containerID="00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc"
May 20 15:54:31.603054 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.603039    2556 scope.go:117] "RemoveContainer" containerID="c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4"
May 20 15:54:31.603300 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:54:31.603283    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4\": container with ID starting with c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4 not found: ID does not exist" containerID="c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4"
May 20 15:54:31.603344 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.603308    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4"} err="failed to get container status \"c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4\": rpc error: code = NotFound desc = could not find container \"c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4\": container with ID starting with c8340bdc53e6c8bb16eec6c8d6a0776a2589ac0bb64b3004497b7ec535b4f6e4 not found: ID does not exist"
May 20 15:54:31.603344 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.603326    2556 scope.go:117] "RemoveContainer" containerID="a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a"
May 20 15:54:31.603559 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:54:31.603539    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a\": container with ID starting with a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a not found: ID does not exist" containerID="a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a"
May 20 15:54:31.603619 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.603568    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a"} err="failed to get container status \"a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a\": rpc error: code = NotFound desc = could not find container \"a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a\": container with ID starting with a3a45d5f8569d6f6f1c3cd860671f6451139cac1f2b3d8cfc01e44454d4bdb5a not found: ID does not exist"
May 20 15:54:31.603619 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.603591    2556 scope.go:117] "RemoveContainer" containerID="00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc"
May 20 15:54:31.603819 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:54:31.603798    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc\": container with ID starting with 00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc not found: ID does not exist" containerID="00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc"
May 20 15:54:31.603858 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.603825    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc"} err="failed to get container status \"00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc\": rpc error: code = NotFound desc = could not find container \"00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc\": container with ID starting with 00767e8f72997ad8ca5e27df5b3c671d42c6c977a67c81fa0964344b45b136cc not found: ID does not exist"
May 20 15:54:31.630960 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.630933    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"]
May 20 15:54:31.635739 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.635722    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-p8k4g"]
May 20 15:54:31.890522 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:31.890447    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" path="/var/lib/kubelet/pods/bc0c5586-15de-4047-bf24-1afff14a6de4/volumes"
May 20 15:54:35.597005 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:35.596967    2556 generic.go:358] "Generic (PLEG): container finished" podID="f4384a77-580e-43b8-868b-6244e69994cc" containerID="a983854a356e0811ffc48c2e539ef879f46806a0837b9b8415a34f0d9487eb49" exitCode=0
May 20 15:54:35.597515 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:35.597009    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerDied","Data":"a983854a356e0811ffc48c2e539ef879f46806a0837b9b8415a34f0d9487eb49"}
May 20 15:54:36.602375 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:36.602336    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerStarted","Data":"a844d2bbd233e7262b2e66c4557fa2334083f6d42bc42a275ce0ab48a5435ff9"}
May 20 15:54:38.610505 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:38.610469    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerStarted","Data":"d1e1d73f62ac833a5f691d222e36820822408ac42ab68341f284486c24c22f89"}
May 20 15:54:38.610505 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:38.610505    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerStarted","Data":"07cc06e19324c2fb2b73c6aa87102a11c5d60da75e33da488bb0c848ea8444c8"}
May 20 15:54:38.611033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:38.610691    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:38.611033 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:38.610713    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:38.634052 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:38.634009    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podStartSLOduration=7.123384886 podStartE2EDuration="9.633996592s" podCreationTimestamp="2026-05-20 15:54:29 +0000 UTC" firstStartedPulling="2026-05-20 15:54:35.655848469 +0000 UTC m=+1318.347104383" lastFinishedPulling="2026-05-20 15:54:38.166460184 +0000 UTC m=+1320.857716089" observedRunningTime="2026-05-20 15:54:38.632020798 +0000 UTC m=+1321.323276727" watchObservedRunningTime="2026-05-20 15:54:38.633996592 +0000 UTC m=+1321.325252519"
May 20 15:54:39.613107 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:39.613063    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:54:45.621030 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:54:45.621002    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:55:05.622266 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:05.622179    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.27:8080: connect: connection refused"
May 20 15:55:15.623733 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:15.623695    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:55:45.624606 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:45.624579    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:55:49.858479 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.858447    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"]
May 20 15:55:49.858880 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.858855    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-container" containerID="cri-o://a844d2bbd233e7262b2e66c4557fa2334083f6d42bc42a275ce0ab48a5435ff9" gracePeriod=30
May 20 15:55:49.858990 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.858948    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy" containerID="cri-o://d1e1d73f62ac833a5f691d222e36820822408ac42ab68341f284486c24c22f89" gracePeriod=30
May 20 15:55:49.858990 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.858951    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-agent" containerID="cri-o://07cc06e19324c2fb2b73c6aa87102a11c5d60da75e33da488bb0c848ea8444c8" gracePeriod=30
May 20 15:55:49.946345 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946314    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"]
May 20 15:55:49.946578 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946566    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="kube-rbac-proxy"
May 20 15:55:49.946622 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946579    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="kube-rbac-proxy"
May 20 15:55:49.946622 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946599    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="storage-initializer"
May 20 15:55:49.946622 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946605    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="storage-initializer"
May 20 15:55:49.946622 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946615    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="kserve-container"
May 20 15:55:49.946622 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946620    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="kserve-container"
May 20 15:55:49.946768 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946659    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="kserve-container"
May 20 15:55:49.946768 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.946666    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc0c5586-15de-4047-bf24-1afff14a6de4" containerName="kube-rbac-proxy"
May 20 15:55:49.949529 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.949513    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:49.952049 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.952032    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
May 20 15:55:49.952116 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.952031    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
May 20 15:55:49.959354 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:49.959332    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"]
May 20 15:55:50.082172 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.082149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpjm\" (UniqueName: \"kubernetes.io/projected/463a0398-0a99-47ab-ad4b-afb239b7d319-kube-api-access-xfpjm\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.082294 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.082184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/463a0398-0a99-47ab-ad4b-afb239b7d319-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.082294 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.082259    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/463a0398-0a99-47ab-ad4b-afb239b7d319-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.082376 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.082336    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/463a0398-0a99-47ab-ad4b-afb239b7d319-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.183486 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.183421    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpjm\" (UniqueName: \"kubernetes.io/projected/463a0398-0a99-47ab-ad4b-afb239b7d319-kube-api-access-xfpjm\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.183486 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.183451    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/463a0398-0a99-47ab-ad4b-afb239b7d319-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.183669 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.183490    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/463a0398-0a99-47ab-ad4b-afb239b7d319-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.183669 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.183547    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/463a0398-0a99-47ab-ad4b-afb239b7d319-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.183899 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.183879    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/463a0398-0a99-47ab-ad4b-afb239b7d319-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.184153 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.184138    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/463a0398-0a99-47ab-ad4b-afb239b7d319-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.185862 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.185843    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/463a0398-0a99-47ab-ad4b-afb239b7d319-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.192292 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.192270    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpjm\" (UniqueName: \"kubernetes.io/projected/463a0398-0a99-47ab-ad4b-afb239b7d319-kube-api-access-xfpjm\") pod \"isvc-paddle-predictor-6b8b7cfb4b-9z5rh\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.259749 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.259726    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:55:50.387016 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.386981    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"]
May 20 15:55:50.389787 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:55:50.389763    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463a0398_0a99_47ab_ad4b_afb239b7d319.slice/crio-fd4806a3f87e8a8a03cd7e23c2f6dd46fdbd961e352c341a911682d4945a7e2a WatchSource:0}: Error finding container fd4806a3f87e8a8a03cd7e23c2f6dd46fdbd961e352c341a911682d4945a7e2a: Status 404 returned error can't find the container with id fd4806a3f87e8a8a03cd7e23c2f6dd46fdbd961e352c341a911682d4945a7e2a
May 20 15:55:50.617064 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.617020    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 20 15:55:50.809207 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.809173    2556 generic.go:358] "Generic (PLEG): container finished" podID="f4384a77-580e-43b8-868b-6244e69994cc" containerID="d1e1d73f62ac833a5f691d222e36820822408ac42ab68341f284486c24c22f89" exitCode=2
May 20 15:55:50.809397 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.809260    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerDied","Data":"d1e1d73f62ac833a5f691d222e36820822408ac42ab68341f284486c24c22f89"}
May 20 15:55:50.810700 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.810662    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" event={"ID":"463a0398-0a99-47ab-ad4b-afb239b7d319","Type":"ContainerStarted","Data":"368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248"}
May 20 15:55:50.810700 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:50.810701    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" event={"ID":"463a0398-0a99-47ab-ad4b-afb239b7d319","Type":"ContainerStarted","Data":"fd4806a3f87e8a8a03cd7e23c2f6dd46fdbd961e352c341a911682d4945a7e2a"}
May 20 15:55:51.815632 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:51.815603    2556 generic.go:358] "Generic (PLEG): container finished" podID="f4384a77-580e-43b8-868b-6244e69994cc" containerID="a844d2bbd233e7262b2e66c4557fa2334083f6d42bc42a275ce0ab48a5435ff9" exitCode=0
May 20 15:55:51.816044 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:51.815678    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerDied","Data":"a844d2bbd233e7262b2e66c4557fa2334083f6d42bc42a275ce0ab48a5435ff9"}
May 20 15:55:55.616725 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:55.616693    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 20 15:55:55.622086 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:55.622062    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.27:8080: connect: connection refused"
May 20 15:55:55.827649 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:55.827621    2556 generic.go:358] "Generic (PLEG): container finished" podID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerID="368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248" exitCode=0
May 20 15:55:55.827765 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:55:55.827691    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" event={"ID":"463a0398-0a99-47ab-ad4b-afb239b7d319","Type":"ContainerDied","Data":"368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248"}
May 20 15:56:00.617487 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:00.617447    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 20 15:56:00.617911 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:00.617599    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:56:05.617410 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:05.617362    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 20 15:56:05.621852 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:05.621825    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.27:8080: connect: connection refused"
May 20 15:56:07.867119 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:07.867011    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" event={"ID":"463a0398-0a99-47ab-ad4b-afb239b7d319","Type":"ContainerStarted","Data":"5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159"}
May 20 15:56:07.867119 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:07.867048    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" event={"ID":"463a0398-0a99-47ab-ad4b-afb239b7d319","Type":"ContainerStarted","Data":"6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca"}
May 20 15:56:07.867580 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:07.867404    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:56:07.867580 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:07.867539    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:56:07.868960 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:07.868933    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 20 15:56:07.888059 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:07.888015    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podStartSLOduration=7.103807202 podStartE2EDuration="18.88800366s" podCreationTimestamp="2026-05-20 15:55:49 +0000 UTC" firstStartedPulling="2026-05-20 15:55:55.828735786 +0000 UTC m=+1398.519991693" lastFinishedPulling="2026-05-20 15:56:07.612932241 +0000 UTC m=+1410.304188151" observedRunningTime="2026-05-20 15:56:07.886642919 +0000 UTC m=+1410.577898848" watchObservedRunningTime="2026-05-20 15:56:07.88800366 +0000 UTC m=+1410.579259634"
May 20 15:56:08.869416 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:08.869368    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 20 15:56:10.616710 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:10.616664    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 20 15:56:13.873418 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:13.873380    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:56:13.873910 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:13.873882    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 20 15:56:15.617482 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:15.617439    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 20 15:56:15.621919 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:15.621889    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.27:8080: connect: connection refused"
May 20 15:56:15.622043 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:15.622029    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:56:19.910027 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:19.909998    2556 generic.go:358] "Generic (PLEG): container finished" podID="f4384a77-580e-43b8-868b-6244e69994cc" containerID="07cc06e19324c2fb2b73c6aa87102a11c5d60da75e33da488bb0c848ea8444c8" exitCode=137
May 20 15:56:19.910440 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:19.910071    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerDied","Data":"07cc06e19324c2fb2b73c6aa87102a11c5d60da75e33da488bb0c848ea8444c8"}
May 20 15:56:20.030987 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.030967    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:56:20.109951 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.109928    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls\") pod \"f4384a77-580e-43b8-868b-6244e69994cc\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") "
May 20 15:56:20.110081 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.109961    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4384a77-580e-43b8-868b-6244e69994cc-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"f4384a77-580e-43b8-868b-6244e69994cc\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") "
May 20 15:56:20.110081 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.110009    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zlg2\" (UniqueName: \"kubernetes.io/projected/f4384a77-580e-43b8-868b-6244e69994cc-kube-api-access-8zlg2\") pod \"f4384a77-580e-43b8-868b-6244e69994cc\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") "
May 20 15:56:20.110081 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.110037    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4384a77-580e-43b8-868b-6244e69994cc-kserve-provision-location\") pod \"f4384a77-580e-43b8-868b-6244e69994cc\" (UID: \"f4384a77-580e-43b8-868b-6244e69994cc\") "
May 20 15:56:20.110341 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.110320    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4384a77-580e-43b8-868b-6244e69994cc-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "f4384a77-580e-43b8-868b-6244e69994cc" (UID: "f4384a77-580e-43b8-868b-6244e69994cc"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:56:20.110495 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.110470    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4384a77-580e-43b8-868b-6244e69994cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4384a77-580e-43b8-868b-6244e69994cc" (UID: "f4384a77-580e-43b8-868b-6244e69994cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:56:20.111990 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.111963    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4384a77-580e-43b8-868b-6244e69994cc-kube-api-access-8zlg2" (OuterVolumeSpecName: "kube-api-access-8zlg2") pod "f4384a77-580e-43b8-868b-6244e69994cc" (UID: "f4384a77-580e-43b8-868b-6244e69994cc"). InnerVolumeSpecName "kube-api-access-8zlg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:56:20.112087 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.112001    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4384a77-580e-43b8-868b-6244e69994cc" (UID: "f4384a77-580e-43b8-868b-6244e69994cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:56:20.210505 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.210480    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlg2\" (UniqueName: \"kubernetes.io/projected/f4384a77-580e-43b8-868b-6244e69994cc-kube-api-access-8zlg2\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:56:20.210505 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.210502    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4384a77-580e-43b8-868b-6244e69994cc-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:56:20.210628 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.210512    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4384a77-580e-43b8-868b-6244e69994cc-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:56:20.210628 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.210521    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4384a77-580e-43b8-868b-6244e69994cc-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:56:20.914408 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.914378    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9" event={"ID":"f4384a77-580e-43b8-868b-6244e69994cc","Type":"ContainerDied","Data":"7f8450f99821dad2c0c1a9e43cd399abb74cd477052e60a34a62f95aae176667"}
May 20 15:56:20.914779 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.914422    2556 scope.go:117] "RemoveContainer" containerID="d1e1d73f62ac833a5f691d222e36820822408ac42ab68341f284486c24c22f89"
May 20 15:56:20.914779 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.914476    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"
May 20 15:56:20.922512 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.922488    2556 scope.go:117] "RemoveContainer" containerID="07cc06e19324c2fb2b73c6aa87102a11c5d60da75e33da488bb0c848ea8444c8"
May 20 15:56:20.929129 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.929113    2556 scope.go:117] "RemoveContainer" containerID="a844d2bbd233e7262b2e66c4557fa2334083f6d42bc42a275ce0ab48a5435ff9"
May 20 15:56:20.935824 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.935807    2556 scope.go:117] "RemoveContainer" containerID="a983854a356e0811ffc48c2e539ef879f46806a0837b9b8415a34f0d9487eb49"
May 20 15:56:20.937550 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.937528    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"]
May 20 15:56:20.942340 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:20.942317    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5dc4749bd-kvcg9"]
May 20 15:56:21.890775 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:21.890731    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4384a77-580e-43b8-868b-6244e69994cc" path="/var/lib/kubelet/pods/f4384a77-580e-43b8-868b-6244e69994cc/volumes"
May 20 15:56:23.874588 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:23.874555    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 20 15:56:33.874202 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:33.874093    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 20 15:56:43.873920 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:43.873882    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 20 15:56:53.874951 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:56:53.874922    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:57:01.593670 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.593626    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"]
May 20 15:57:01.594182 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.593959    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container" containerID="cri-o://6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca" gracePeriod=30
May 20 15:57:01.594182 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.593984    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kube-rbac-proxy" containerID="cri-o://5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159" gracePeriod=30
May 20 15:57:01.666222 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666170    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"]
May 20 15:57:01.666512 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666496    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="storage-initializer"
May 20 15:57:01.666590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666515    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="storage-initializer"
May 20 15:57:01.666590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666534    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-agent"
May 20 15:57:01.666590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666543    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-agent"
May 20 15:57:01.666590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666559    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy"
May 20 15:57:01.666590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666568    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy"
May 20 15:57:01.666590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666583    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-container"
May 20 15:57:01.666590 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666592    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-container"
May 20 15:57:01.666923 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666653    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-container"
May 20 15:57:01.666923 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666666    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kube-rbac-proxy"
May 20 15:57:01.666923 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.666677    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="f4384a77-580e-43b8-868b-6244e69994cc" containerName="kserve-agent"
May 20 15:57:01.676382 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.676359    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.678325 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.678303    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"]
May 20 15:57:01.678768 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.678748    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
May 20 15:57:01.678850 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.678765    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
May 20 15:57:01.797703 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.797668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.797880 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.797712    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab4bd3cd-676f-485c-b945-3d6d866da7bc-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.797880 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.797744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2dj\" (UniqueName: \"kubernetes.io/projected/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kube-api-access-7w2dj\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.797880 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.797783    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab4bd3cd-676f-485c-b945-3d6d866da7bc-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.899089 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.899016    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab4bd3cd-676f-485c-b945-3d6d866da7bc-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.899089 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.899051    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2dj\" (UniqueName: \"kubernetes.io/projected/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kube-api-access-7w2dj\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.899089 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.899072    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab4bd3cd-676f-485c-b945-3d6d866da7bc-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.899445 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.899142    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.899522 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.899504    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.899766 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.899742    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab4bd3cd-676f-485c-b945-3d6d866da7bc-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.901484 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.901466    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab4bd3cd-676f-485c-b945-3d6d866da7bc-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.908225 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.908207    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2dj\" (UniqueName: \"kubernetes.io/projected/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kube-api-access-7w2dj\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:01.986845 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:01.986814    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:02.021635 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:02.021606    2556 generic.go:358] "Generic (PLEG): container finished" podID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerID="5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159" exitCode=2
May 20 15:57:02.021777 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:02.021655    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" event={"ID":"463a0398-0a99-47ab-ad4b-afb239b7d319","Type":"ContainerDied","Data":"5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159"}
May 20 15:57:02.114864 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:02.114838    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"]
May 20 15:57:02.117655 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:57:02.117627    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4bd3cd_676f_485c_b945_3d6d866da7bc.slice/crio-295f09e9bb265dcdce8a31cf3aec28e412e13f67458f0cdb784c2328dc82bb5e WatchSource:0}: Error finding container 295f09e9bb265dcdce8a31cf3aec28e412e13f67458f0cdb784c2328dc82bb5e: Status 404 returned error can't find the container with id 295f09e9bb265dcdce8a31cf3aec28e412e13f67458f0cdb784c2328dc82bb5e
May 20 15:57:03.025165 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:03.025128    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" event={"ID":"ab4bd3cd-676f-485c-b945-3d6d866da7bc","Type":"ContainerStarted","Data":"090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34"}
May 20 15:57:03.025165 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:03.025164    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" event={"ID":"ab4bd3cd-676f-485c-b945-3d6d866da7bc","Type":"ContainerStarted","Data":"295f09e9bb265dcdce8a31cf3aec28e412e13f67458f0cdb784c2328dc82bb5e"}
May 20 15:57:03.870198 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:03.870160    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
May 20 15:57:03.873810 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:03.873786    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 20 15:57:04.240599 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.240578    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:57:04.319224 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.319197    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/463a0398-0a99-47ab-ad4b-afb239b7d319-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"463a0398-0a99-47ab-ad4b-afb239b7d319\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") "
May 20 15:57:04.319354 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.319249    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/463a0398-0a99-47ab-ad4b-afb239b7d319-kserve-provision-location\") pod \"463a0398-0a99-47ab-ad4b-afb239b7d319\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") "
May 20 15:57:04.319354 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.319283    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/463a0398-0a99-47ab-ad4b-afb239b7d319-proxy-tls\") pod \"463a0398-0a99-47ab-ad4b-afb239b7d319\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") "
May 20 15:57:04.319354 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.319314    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfpjm\" (UniqueName: \"kubernetes.io/projected/463a0398-0a99-47ab-ad4b-afb239b7d319-kube-api-access-xfpjm\") pod \"463a0398-0a99-47ab-ad4b-afb239b7d319\" (UID: \"463a0398-0a99-47ab-ad4b-afb239b7d319\") "
May 20 15:57:04.319586 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.319568    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463a0398-0a99-47ab-ad4b-afb239b7d319-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "463a0398-0a99-47ab-ad4b-afb239b7d319" (UID: "463a0398-0a99-47ab-ad4b-afb239b7d319"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:57:04.321424 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.321390    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463a0398-0a99-47ab-ad4b-afb239b7d319-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "463a0398-0a99-47ab-ad4b-afb239b7d319" (UID: "463a0398-0a99-47ab-ad4b-afb239b7d319"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:57:04.321561 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.321545    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463a0398-0a99-47ab-ad4b-afb239b7d319-kube-api-access-xfpjm" (OuterVolumeSpecName: "kube-api-access-xfpjm") pod "463a0398-0a99-47ab-ad4b-afb239b7d319" (UID: "463a0398-0a99-47ab-ad4b-afb239b7d319"). InnerVolumeSpecName "kube-api-access-xfpjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:57:04.329000 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.328980    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463a0398-0a99-47ab-ad4b-afb239b7d319-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "463a0398-0a99-47ab-ad4b-afb239b7d319" (UID: "463a0398-0a99-47ab-ad4b-afb239b7d319"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:57:04.420469 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.420414    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/463a0398-0a99-47ab-ad4b-afb239b7d319-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:57:04.420469 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.420436    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/463a0398-0a99-47ab-ad4b-afb239b7d319-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:57:04.420469 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.420446    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/463a0398-0a99-47ab-ad4b-afb239b7d319-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:57:04.420469 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:04.420454    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfpjm\" (UniqueName: \"kubernetes.io/projected/463a0398-0a99-47ab-ad4b-afb239b7d319-kube-api-access-xfpjm\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:57:05.031346 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.031312    2556 generic.go:358] "Generic (PLEG): container finished" podID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerID="6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca" exitCode=0
May 20 15:57:05.031510 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.031354    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" event={"ID":"463a0398-0a99-47ab-ad4b-afb239b7d319","Type":"ContainerDied","Data":"6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca"}
May 20 15:57:05.031510 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.031382    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh" event={"ID":"463a0398-0a99-47ab-ad4b-afb239b7d319","Type":"ContainerDied","Data":"fd4806a3f87e8a8a03cd7e23c2f6dd46fdbd961e352c341a911682d4945a7e2a"}
May 20 15:57:05.031510 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.031401    2556 scope.go:117] "RemoveContainer" containerID="5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159"
May 20 15:57:05.031510 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.031403    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"
May 20 15:57:05.039191 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.039076    2556 scope.go:117] "RemoveContainer" containerID="6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca"
May 20 15:57:05.045630 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.045612    2556 scope.go:117] "RemoveContainer" containerID="368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248"
May 20 15:57:05.051727 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.051711    2556 scope.go:117] "RemoveContainer" containerID="5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159"
May 20 15:57:05.052058 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:57:05.052030    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159\": container with ID starting with 5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159 not found: ID does not exist" containerID="5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159"
May 20 15:57:05.052186 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.052070    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159"} err="failed to get container status \"5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159\": rpc error: code = NotFound desc = could not find container \"5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159\": container with ID starting with 5415c217feff74ced2b82b57f5c54c7a38f73e4ca07cf2a8e1ffa3d5eee7d159 not found: ID does not exist"
May 20 15:57:05.052186 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.052109    2556 scope.go:117] "RemoveContainer" containerID="6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca"
May 20 15:57:05.052441 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:57:05.052370    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca\": container with ID starting with 6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca not found: ID does not exist" containerID="6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca"
May 20 15:57:05.052441 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.052406    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca"} err="failed to get container status \"6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca\": rpc error: code = NotFound desc = could not find container \"6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca\": container with ID starting with 6e84aeb2599621fb9dac733faf0320b5e3c74d59b9d83aebe920f1882131adca not found: ID does not exist"
May 20 15:57:05.052441 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.052428    2556 scope.go:117] "RemoveContainer" containerID="368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248"
May 20 15:57:05.052754 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:57:05.052735    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248\": container with ID starting with 368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248 not found: ID does not exist" containerID="368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248"
May 20 15:57:05.052805 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.052760    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248"} err="failed to get container status \"368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248\": rpc error: code = NotFound desc = could not find container \"368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248\": container with ID starting with 368b5b2e8cade200ab3bd5a49417f3a1285b12f6581847a383c49cfbbd152248 not found: ID does not exist"
May 20 15:57:05.053640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.053623    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"]
May 20 15:57:05.059895 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.058165    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-9z5rh"]
May 20 15:57:05.891556 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:05.891523    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" path="/var/lib/kubelet/pods/463a0398-0a99-47ab-ad4b-afb239b7d319/volumes"
May 20 15:57:07.038134 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:07.038080    2556 generic.go:358] "Generic (PLEG): container finished" podID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerID="090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34" exitCode=0
May 20 15:57:07.038578 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:07.038160    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" event={"ID":"ab4bd3cd-676f-485c-b945-3d6d866da7bc","Type":"ContainerDied","Data":"090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34"}
May 20 15:57:08.042196 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:08.042167    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" event={"ID":"ab4bd3cd-676f-485c-b945-3d6d866da7bc","Type":"ContainerStarted","Data":"884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549"}
May 20 15:57:08.042568 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:08.042201    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" event={"ID":"ab4bd3cd-676f-485c-b945-3d6d866da7bc","Type":"ContainerStarted","Data":"9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76"}
May 20 15:57:08.042568 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:08.042379    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:08.062313 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:08.062273    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podStartSLOduration=7.0622626 podStartE2EDuration="7.0622626s" podCreationTimestamp="2026-05-20 15:57:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:57:08.061211339 +0000 UTC m=+1470.752467258" watchObservedRunningTime="2026-05-20 15:57:08.0622626 +0000 UTC m=+1470.753518592"
May 20 15:57:09.045460 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:09.045429    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:09.046350 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:09.046324    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 20 15:57:10.048342 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:10.048309    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 20 15:57:15.052443 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:15.052404    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:57:15.052871 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:15.052845    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 20 15:57:25.053335 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:25.053294    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 20 15:57:35.053068 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:35.053028    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 20 15:57:45.052962 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:45.052921    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 20 15:57:55.053276 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:57:55.053241    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:58:03.374028 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.373945    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"]
May 20 15:58:03.374489 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.374279    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container" containerID="cri-o://9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76" gracePeriod=30
May 20 15:58:03.374489 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.374335    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kube-rbac-proxy" containerID="cri-o://884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549" gracePeriod=30
May 20 15:58:03.482978 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.482950    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"]
May 20 15:58:03.483253 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.483235    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container"
May 20 15:58:03.483253 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.483251    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container"
May 20 15:58:03.483425 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.483264    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kube-rbac-proxy"
May 20 15:58:03.483425 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.483269    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kube-rbac-proxy"
May 20 15:58:03.483425 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.483279    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="storage-initializer"
May 20 15:58:03.483425 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.483285    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="storage-initializer"
May 20 15:58:03.483425 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.483329    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kube-rbac-proxy"
May 20 15:58:03.483425 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.483338    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="463a0398-0a99-47ab-ad4b-afb239b7d319" containerName="kserve-container"
May 20 15:58:03.487667 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.487650    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.490186 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.490166    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
May 20 15:58:03.490292 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.490190    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
May 20 15:58:03.498121 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.498073    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"]
May 20 15:58:03.615916 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.615883    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d531b95d-2607-4483-8e39-e3a6c5354a02-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.616057 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.615932    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d227c\" (UniqueName: \"kubernetes.io/projected/d531b95d-2607-4483-8e39-e3a6c5354a02-kube-api-access-d227c\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.616057 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.615995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.616057 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.616033    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d531b95d-2607-4483-8e39-e3a6c5354a02-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.716420 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.716357    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d227c\" (UniqueName: \"kubernetes.io/projected/d531b95d-2607-4483-8e39-e3a6c5354a02-kube-api-access-d227c\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.716420 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.716391    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.716420 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.716416    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d531b95d-2607-4483-8e39-e3a6c5354a02-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.716597 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:58:03.716502    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
May 20 15:58:03.716597 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:58:03.716565    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls podName:d531b95d-2607-4483-8e39-e3a6c5354a02 nodeName:}" failed. No retries permitted until 2026-05-20 15:58:04.216549433 +0000 UTC m=+1526.907805338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" (UID: "d531b95d-2607-4483-8e39-e3a6c5354a02") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
May 20 15:58:03.716719 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.716503    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d531b95d-2607-4483-8e39-e3a6c5354a02-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.716917 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.716896    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d531b95d-2607-4483-8e39-e3a6c5354a02-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.717060 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.717045    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d531b95d-2607-4483-8e39-e3a6c5354a02-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:03.725684 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:03.725655    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d227c\" (UniqueName: \"kubernetes.io/projected/d531b95d-2607-4483-8e39-e3a6c5354a02-kube-api-access-d227c\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:04.188473 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:04.188438    2556 generic.go:358] "Generic (PLEG): container finished" podID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerID="884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549" exitCode=2
May 20 15:58:04.188632 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:04.188496    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" event={"ID":"ab4bd3cd-676f-485c-b945-3d6d866da7bc","Type":"ContainerDied","Data":"884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549"}
May 20 15:58:04.220018 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:04.219993    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:04.222547 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:04.222517    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:04.397228 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:04.397192    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:04.519665 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:04.519642    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"]
May 20 15:58:04.522491 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:58:04.522459    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd531b95d_2607_4483_8e39_e3a6c5354a02.slice/crio-747d5ec46a53e64df7745ec42590cbbea82bb064509452bcf8b7623620542cef WatchSource:0}: Error finding container 747d5ec46a53e64df7745ec42590cbbea82bb064509452bcf8b7623620542cef: Status 404 returned error can't find the container with id 747d5ec46a53e64df7745ec42590cbbea82bb064509452bcf8b7623620542cef
May 20 15:58:04.524272 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:04.524257    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:58:05.048759 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:05.048720    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
May 20 15:58:05.053041 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:05.053016    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 20 15:58:05.192744 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:05.192709    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" event={"ID":"d531b95d-2607-4483-8e39-e3a6c5354a02","Type":"ContainerStarted","Data":"0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70"}
May 20 15:58:05.192744 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:05.192745    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" event={"ID":"d531b95d-2607-4483-8e39-e3a6c5354a02","Type":"ContainerStarted","Data":"747d5ec46a53e64df7745ec42590cbbea82bb064509452bcf8b7623620542cef"}
May 20 15:58:06.013718 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.013696    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:58:06.131747 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.131690    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab4bd3cd-676f-485c-b945-3d6d866da7bc-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") "
May 20 15:58:06.131869 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.131774    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w2dj\" (UniqueName: \"kubernetes.io/projected/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kube-api-access-7w2dj\") pod \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") "
May 20 15:58:06.131940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.131871    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab4bd3cd-676f-485c-b945-3d6d866da7bc-proxy-tls\") pod \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") "
May 20 15:58:06.131940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.131920    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kserve-provision-location\") pod \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\" (UID: \"ab4bd3cd-676f-485c-b945-3d6d866da7bc\") "
May 20 15:58:06.132076 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.132051    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4bd3cd-676f-485c-b945-3d6d866da7bc-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "ab4bd3cd-676f-485c-b945-3d6d866da7bc" (UID: "ab4bd3cd-676f-485c-b945-3d6d866da7bc"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:58:06.132282 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.132266    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab4bd3cd-676f-485c-b945-3d6d866da7bc-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:58:06.133679 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.133658    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kube-api-access-7w2dj" (OuterVolumeSpecName: "kube-api-access-7w2dj") pod "ab4bd3cd-676f-485c-b945-3d6d866da7bc" (UID: "ab4bd3cd-676f-485c-b945-3d6d866da7bc"). InnerVolumeSpecName "kube-api-access-7w2dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:58:06.133887 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.133871    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4bd3cd-676f-485c-b945-3d6d866da7bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab4bd3cd-676f-485c-b945-3d6d866da7bc" (UID: "ab4bd3cd-676f-485c-b945-3d6d866da7bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:58:06.146545 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.146519    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab4bd3cd-676f-485c-b945-3d6d866da7bc" (UID: "ab4bd3cd-676f-485c-b945-3d6d866da7bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:06.197158 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.197132    2556 generic.go:358] "Generic (PLEG): container finished" podID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerID="9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76" exitCode=0
May 20 15:58:06.197253 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.197211    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"
May 20 15:58:06.197253 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.197217    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" event={"ID":"ab4bd3cd-676f-485c-b945-3d6d866da7bc","Type":"ContainerDied","Data":"9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76"}
May 20 15:58:06.197328 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.197258    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh" event={"ID":"ab4bd3cd-676f-485c-b945-3d6d866da7bc","Type":"ContainerDied","Data":"295f09e9bb265dcdce8a31cf3aec28e412e13f67458f0cdb784c2328dc82bb5e"}
May 20 15:58:06.197328 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.197278    2556 scope.go:117] "RemoveContainer" containerID="884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549"
May 20 15:58:06.205192 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.205176    2556 scope.go:117] "RemoveContainer" containerID="9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76"
May 20 15:58:06.211532 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.211518    2556 scope.go:117] "RemoveContainer" containerID="090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34"
May 20 15:58:06.217457 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.217440    2556 scope.go:117] "RemoveContainer" containerID="884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549"
May 20 15:58:06.217687 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:58:06.217673    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549\": container with ID starting with 884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549 not found: ID does not exist" containerID="884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549"
May 20 15:58:06.217726 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.217694    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549"} err="failed to get container status \"884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549\": rpc error: code = NotFound desc = could not find container \"884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549\": container with ID starting with 884a3b5aa046d452bfad896fa57f7663789194499c5cf8ea241d123a0c0ff549 not found: ID does not exist"
May 20 15:58:06.217726 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.217710    2556 scope.go:117] "RemoveContainer" containerID="9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76"
May 20 15:58:06.217904 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:58:06.217890    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76\": container with ID starting with 9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76 not found: ID does not exist" containerID="9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76"
May 20 15:58:06.217940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.217908    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76"} err="failed to get container status \"9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76\": rpc error: code = NotFound desc = could not find container \"9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76\": container with ID starting with 9829887dc5384f02fd27c50a3b91e2f0f5d4065fb0f5f78c4a94dfdcb694df76 not found: ID does not exist"
May 20 15:58:06.217940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.217921    2556 scope.go:117] "RemoveContainer" containerID="090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34"
May 20 15:58:06.218167 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:58:06.218084    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34\": container with ID starting with 090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34 not found: ID does not exist" containerID="090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34"
May 20 15:58:06.218167 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.218117    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34"} err="failed to get container status \"090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34\": rpc error: code = NotFound desc = could not find container \"090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34\": container with ID starting with 090769cb8ded36b7d34f80a88eff0c9937557ea09647ee1896839d346a06ca34 not found: ID does not exist"
May 20 15:58:06.220710 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.220691    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"]
May 20 15:58:06.224179 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.224162    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-zzqgh"]
May 20 15:58:06.233177 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.233162    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w2dj\" (UniqueName: \"kubernetes.io/projected/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kube-api-access-7w2dj\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:58:06.233234 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.233178    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab4bd3cd-676f-485c-b945-3d6d866da7bc-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:58:06.233234 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:06.233188    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4bd3cd-676f-485c-b945-3d6d866da7bc-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:58:07.891686 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:07.891645    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" path="/var/lib/kubelet/pods/ab4bd3cd-676f-485c-b945-3d6d866da7bc/volumes"
May 20 15:58:09.209832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:09.209737    2556 generic.go:358] "Generic (PLEG): container finished" podID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerID="0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70" exitCode=0
May 20 15:58:09.209832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:09.209814    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" event={"ID":"d531b95d-2607-4483-8e39-e3a6c5354a02","Type":"ContainerDied","Data":"0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70"}
May 20 15:58:10.214832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:10.214793    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" event={"ID":"d531b95d-2607-4483-8e39-e3a6c5354a02","Type":"ContainerStarted","Data":"1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680"}
May 20 15:58:10.214832 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:10.214834    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" event={"ID":"d531b95d-2607-4483-8e39-e3a6c5354a02","Type":"ContainerStarted","Data":"6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5"}
May 20 15:58:10.215519 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:10.215138    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:10.215519 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:10.215266    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:10.216400 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:10.216372    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 20 15:58:10.236168 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:10.236126    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podStartSLOduration=7.236089054 podStartE2EDuration="7.236089054s" podCreationTimestamp="2026-05-20 15:58:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:58:10.234702141 +0000 UTC m=+1532.925958069" watchObservedRunningTime="2026-05-20 15:58:10.236089054 +0000 UTC m=+1532.927344983"
May 20 15:58:11.218339 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:11.218292    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 20 15:58:16.222591 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:16.222558    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:58:16.223111 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:16.223072    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 20 15:58:26.223652 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:26.223611    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 20 15:58:36.223701 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:36.223665    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 20 15:58:46.223310 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:46.223275    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 20 15:58:56.223793 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:58:56.223762    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:59:05.180940 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.180908    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"]
May 20 15:59:05.181375 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.181347    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container" containerID="cri-o://6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5" gracePeriod=30
May 20 15:59:05.181447 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.181373    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kube-rbac-proxy" containerID="cri-o://1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680" gracePeriod=30
May 20 15:59:05.250243 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250212    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"]
May 20 15:59:05.250514 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250498    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="storage-initializer"
May 20 15:59:05.250588 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250518    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="storage-initializer"
May 20 15:59:05.250588 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250533    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kube-rbac-proxy"
May 20 15:59:05.250588 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250542    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kube-rbac-proxy"
May 20 15:59:05.250588 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250571    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container"
May 20 15:59:05.250588 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250580    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container"
May 20 15:59:05.250827 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250644    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kube-rbac-proxy"
May 20 15:59:05.250827 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.250657    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="ab4bd3cd-676f-485c-b945-3d6d866da7bc" containerName="kserve-container"
May 20 15:59:05.254903 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.254885    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.257248 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.257225    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
May 20 15:59:05.257248 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.257239    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
May 20 15:59:05.263300 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.263279    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"]
May 20 15:59:05.324602 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.324577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vt7\" (UniqueName: \"kubernetes.io/projected/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kube-api-access-79vt7\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.324690 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.324608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.324690 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.324629    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.324690 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.324682    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.366418 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.366396    2556 generic.go:358] "Generic (PLEG): container finished" podID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerID="1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680" exitCode=2
May 20 15:59:05.366514 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.366468    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" event={"ID":"d531b95d-2607-4483-8e39-e3a6c5354a02","Type":"ContainerDied","Data":"1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680"}
May 20 15:59:05.425320 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.425293    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-79vt7\" (UniqueName: \"kubernetes.io/projected/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kube-api-access-79vt7\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.425442 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.425334    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.425442 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.425365    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.425442 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.425399    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.425596 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:59:05.425578    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
May 20 15:59:05.425721 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:59:05.425708    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls podName:9c46e7b8-dbbe-45ec-820e-fea6e25250ef nodeName:}" failed. No retries permitted until 2026-05-20 15:59:05.925690792 +0000 UTC m=+1588.616946698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls") pod "isvc-pmml-predictor-8bb578669-mvg7t" (UID: "9c46e7b8-dbbe-45ec-820e-fea6e25250ef") : secret "isvc-pmml-predictor-serving-cert" not found
May 20 15:59:05.425800 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.425740    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.425989 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.425971    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.434324 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.434281    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vt7\" (UniqueName: \"kubernetes.io/projected/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kube-api-access-79vt7\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.928441 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.928411    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:05.930729 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:05.930704    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-mvg7t\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:06.166067 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:06.166019    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:06.219498 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:06.219446    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
May 20 15:59:06.223875 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:06.223836    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
May 20 15:59:06.288110 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:06.288063    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"]
May 20 15:59:06.291295 ip-10-0-134-233 kubenswrapper[2556]: W0520 15:59:06.291269    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c46e7b8_dbbe_45ec_820e_fea6e25250ef.slice/crio-f95d8a8d0ad6d6007d0700a3c2b37ea932e21c135ab23daccc2019deb466daaa WatchSource:0}: Error finding container f95d8a8d0ad6d6007d0700a3c2b37ea932e21c135ab23daccc2019deb466daaa: Status 404 returned error can't find the container with id f95d8a8d0ad6d6007d0700a3c2b37ea932e21c135ab23daccc2019deb466daaa
May 20 15:59:06.370346 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:06.370319    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" event={"ID":"9c46e7b8-dbbe-45ec-820e-fea6e25250ef","Type":"ContainerStarted","Data":"d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767"}
May 20 15:59:06.370502 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:06.370350    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" event={"ID":"9c46e7b8-dbbe-45ec-820e-fea6e25250ef","Type":"ContainerStarted","Data":"f95d8a8d0ad6d6007d0700a3c2b37ea932e21c135ab23daccc2019deb466daaa"}
May 20 15:59:07.617443 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.617414    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:59:07.741447 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.741410    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls\") pod \"d531b95d-2607-4483-8e39-e3a6c5354a02\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") "
May 20 15:59:07.741605 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.741466    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d531b95d-2607-4483-8e39-e3a6c5354a02-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"d531b95d-2607-4483-8e39-e3a6c5354a02\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") "
May 20 15:59:07.741605 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.741513    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d531b95d-2607-4483-8e39-e3a6c5354a02-kserve-provision-location\") pod \"d531b95d-2607-4483-8e39-e3a6c5354a02\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") "
May 20 15:59:07.741605 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.741539    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d227c\" (UniqueName: \"kubernetes.io/projected/d531b95d-2607-4483-8e39-e3a6c5354a02-kube-api-access-d227c\") pod \"d531b95d-2607-4483-8e39-e3a6c5354a02\" (UID: \"d531b95d-2607-4483-8e39-e3a6c5354a02\") "
May 20 15:59:07.741844 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.741821    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d531b95d-2607-4483-8e39-e3a6c5354a02-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "d531b95d-2607-4483-8e39-e3a6c5354a02" (UID: "d531b95d-2607-4483-8e39-e3a6c5354a02"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:59:07.743640 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.743610    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d531b95d-2607-4483-8e39-e3a6c5354a02-kube-api-access-d227c" (OuterVolumeSpecName: "kube-api-access-d227c") pod "d531b95d-2607-4483-8e39-e3a6c5354a02" (UID: "d531b95d-2607-4483-8e39-e3a6c5354a02"). InnerVolumeSpecName "kube-api-access-d227c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:59:07.743744 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.743638    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d531b95d-2607-4483-8e39-e3a6c5354a02" (UID: "d531b95d-2607-4483-8e39-e3a6c5354a02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:59:07.751662 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.751640    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d531b95d-2607-4483-8e39-e3a6c5354a02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d531b95d-2607-4483-8e39-e3a6c5354a02" (UID: "d531b95d-2607-4483-8e39-e3a6c5354a02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:59:07.846723 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.846696    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d531b95d-2607-4483-8e39-e3a6c5354a02-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:59:07.846723 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.846723    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d531b95d-2607-4483-8e39-e3a6c5354a02-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:59:07.846870 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.846734    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d227c\" (UniqueName: \"kubernetes.io/projected/d531b95d-2607-4483-8e39-e3a6c5354a02-kube-api-access-d227c\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:59:07.846870 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:07.846744    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d531b95d-2607-4483-8e39-e3a6c5354a02-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 15:59:08.377358 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.377328    2556 generic.go:358] "Generic (PLEG): container finished" podID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerID="6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5" exitCode=0
May 20 15:59:08.377545 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.377410    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" event={"ID":"d531b95d-2607-4483-8e39-e3a6c5354a02","Type":"ContainerDied","Data":"6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5"}
May 20 15:59:08.377545 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.377441    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"
May 20 15:59:08.377545 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.377466    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk" event={"ID":"d531b95d-2607-4483-8e39-e3a6c5354a02","Type":"ContainerDied","Data":"747d5ec46a53e64df7745ec42590cbbea82bb064509452bcf8b7623620542cef"}
May 20 15:59:08.377545 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.377484    2556 scope.go:117] "RemoveContainer" containerID="1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680"
May 20 15:59:08.385204 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.385186    2556 scope.go:117] "RemoveContainer" containerID="6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5"
May 20 15:59:08.391789 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.391772    2556 scope.go:117] "RemoveContainer" containerID="0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70"
May 20 15:59:08.397363 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.397341    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"]
May 20 15:59:08.398680 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.398665    2556 scope.go:117] "RemoveContainer" containerID="1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680"
May 20 15:59:08.398905 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:59:08.398889    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680\": container with ID starting with 1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680 not found: ID does not exist" containerID="1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680"
May 20 15:59:08.398952 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.398911    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680"} err="failed to get container status \"1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680\": rpc error: code = NotFound desc = could not find container \"1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680\": container with ID starting with 1dbe2db925fd1634acbde2940d72bf7039094030e79fafaf3f34a3b4052a0680 not found: ID does not exist"
May 20 15:59:08.398952 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.398928    2556 scope.go:117] "RemoveContainer" containerID="6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5"
May 20 15:59:08.399137 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:59:08.399122    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5\": container with ID starting with 6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5 not found: ID does not exist" containerID="6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5"
May 20 15:59:08.399178 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.399140    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5"} err="failed to get container status \"6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5\": rpc error: code = NotFound desc = could not find container \"6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5\": container with ID starting with 6548c0ab34de21dc78700966ab0873d6cf91b097e8d1f7bd01ad9ab05ca647d5 not found: ID does not exist"
May 20 15:59:08.399178 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.399153    2556 scope.go:117] "RemoveContainer" containerID="0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70"
May 20 15:59:08.399386 ip-10-0-134-233 kubenswrapper[2556]: E0520 15:59:08.399369    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70\": container with ID starting with 0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70 not found: ID does not exist" containerID="0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70"
May 20 15:59:08.399428 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.399389    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70"} err="failed to get container status \"0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70\": rpc error: code = NotFound desc = could not find container \"0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70\": container with ID starting with 0e655ebd507455868cf84e1fdcbd51f74d8930d5b5d70f975a7c30eeb76a4d70 not found: ID does not exist"
May 20 15:59:08.401796 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:08.401744    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-vdnbk"]
May 20 15:59:09.890796 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:09.890759    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" path="/var/lib/kubelet/pods/d531b95d-2607-4483-8e39-e3a6c5354a02/volumes"
May 20 15:59:10.384798 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:10.384761    2556 generic.go:358] "Generic (PLEG): container finished" podID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerID="d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767" exitCode=0
May 20 15:59:10.384964 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:10.384839    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" event={"ID":"9c46e7b8-dbbe-45ec-820e-fea6e25250ef","Type":"ContainerDied","Data":"d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767"}
May 20 15:59:18.413755 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:18.413724    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" event={"ID":"9c46e7b8-dbbe-45ec-820e-fea6e25250ef","Type":"ContainerStarted","Data":"c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83"}
May 20 15:59:18.413755 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:18.413758    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" event={"ID":"9c46e7b8-dbbe-45ec-820e-fea6e25250ef","Type":"ContainerStarted","Data":"8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105"}
May 20 15:59:18.414225 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:18.413946    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:18.434290 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:18.434239    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podStartSLOduration=6.372972788 podStartE2EDuration="13.434225895s" podCreationTimestamp="2026-05-20 15:59:05 +0000 UTC" firstStartedPulling="2026-05-20 15:59:10.386195912 +0000 UTC m=+1593.077451817" lastFinishedPulling="2026-05-20 15:59:17.447449016 +0000 UTC m=+1600.138704924" observedRunningTime="2026-05-20 15:59:18.433170391 +0000 UTC m=+1601.124426320" watchObservedRunningTime="2026-05-20 15:59:18.434225895 +0000 UTC m=+1601.125481822"
May 20 15:59:19.416567 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:19.416533    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:19.417787 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:19.417764    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 15:59:20.418532 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:20.418483    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 15:59:25.422576 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:25.422545    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 15:59:25.423202 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:25.423171    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 15:59:35.423159 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:35.423054    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 15:59:45.423233 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:45.423192    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 15:59:55.423992 ip-10-0-134-233 kubenswrapper[2556]: I0520 15:59:55.423951    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 16:00:05.423285 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:05.423249    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 16:00:15.423740 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:15.423699    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 16:00:25.423583 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:25.423542    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 20 16:00:31.891302 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:31.891264    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 16:00:36.473674 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.473640    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"]
May 20 16:00:36.474091 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.474042    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container" containerID="cri-o://8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105" gracePeriod=30
May 20 16:00:36.474182 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.474076    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kube-rbac-proxy" containerID="cri-o://c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83" gracePeriod=30
May 20 16:00:36.584326 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584300    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"]
May 20 16:00:36.584580 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584566    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="storage-initializer"
May 20 16:00:36.584580 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584580    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="storage-initializer"
May 20 16:00:36.584678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584587    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container"
May 20 16:00:36.584678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584593    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container"
May 20 16:00:36.584678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584604    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kube-rbac-proxy"
May 20 16:00:36.584678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584609    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kube-rbac-proxy"
May 20 16:00:36.584678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584647    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kube-rbac-proxy"
May 20 16:00:36.584678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.584655    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="d531b95d-2607-4483-8e39-e3a6c5354a02" containerName="kserve-container"
May 20 16:00:36.587777 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.587759    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.589942 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.589921    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
May 20 16:00:36.590045 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.590005    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
May 20 16:00:36.595673 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.595655    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"]
May 20 16:00:36.613965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.613938    2556 generic.go:358] "Generic (PLEG): container finished" podID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerID="c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83" exitCode=2
May 20 16:00:36.614053 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.613963    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" event={"ID":"9c46e7b8-dbbe-45ec-820e-fea6e25250ef","Type":"ContainerDied","Data":"c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83"}
May 20 16:00:36.663554 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.663529    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64krj\" (UniqueName: \"kubernetes.io/projected/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kube-api-access-64krj\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.663638 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.663624    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.663697 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.663653    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10881fdf-ac40-4d21-b924-0a4cc1960dcf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.663697 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.663686    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.764809 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.764787    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.764896 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.764826    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64krj\" (UniqueName: \"kubernetes.io/projected/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kube-api-access-64krj\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.764896 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.764891    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.765012 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.764918    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10881fdf-ac40-4d21-b924-0a4cc1960dcf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.765012 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:00:36.764948    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
May 20 16:00:36.765012 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:00:36.765009    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls podName:10881fdf-ac40-4d21-b924-0a4cc1960dcf nodeName:}" failed. No retries permitted until 2026-05-20 16:00:37.264992323 +0000 UTC m=+1679.956248230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls") pod "isvc-pmml-runtime-predictor-67bc544947-7qfgt" (UID: "10881fdf-ac40-4d21-b924-0a4cc1960dcf") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
May 20 16:00:36.765290 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.765269    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.765587 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.765572    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10881fdf-ac40-4d21-b924-0a4cc1960dcf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:36.773761 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:36.773745    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64krj\" (UniqueName: \"kubernetes.io/projected/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kube-api-access-64krj\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:37.268539 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:37.268515    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:37.270922 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:37.270901    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-7qfgt\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:37.498416 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:37.498381    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:37.622052 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:37.622021    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"]
May 20 16:00:37.625042 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:00:37.625010    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10881fdf_ac40_4d21_b924_0a4cc1960dcf.slice/crio-33e07bc1958abf5f3d7064777e9a0ccdc245fe694f05d55fe5610be3ae0815de WatchSource:0}: Error finding container 33e07bc1958abf5f3d7064777e9a0ccdc245fe694f05d55fe5610be3ae0815de: Status 404 returned error can't find the container with id 33e07bc1958abf5f3d7064777e9a0ccdc245fe694f05d55fe5610be3ae0815de
May 20 16:00:38.621597 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:38.621558    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" event={"ID":"10881fdf-ac40-4d21-b924-0a4cc1960dcf","Type":"ContainerStarted","Data":"5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc"}
May 20 16:00:38.621970 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:38.621603    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" event={"ID":"10881fdf-ac40-4d21-b924-0a4cc1960dcf","Type":"ContainerStarted","Data":"33e07bc1958abf5f3d7064777e9a0ccdc245fe694f05d55fe5610be3ae0815de"}
May 20 16:00:39.614088 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.614069    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 16:00:39.624986 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.624960    2556 generic.go:358] "Generic (PLEG): container finished" podID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerID="8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105" exitCode=0
May 20 16:00:39.625362 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.625021    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"
May 20 16:00:39.625362 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.625049    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" event={"ID":"9c46e7b8-dbbe-45ec-820e-fea6e25250ef","Type":"ContainerDied","Data":"8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105"}
May 20 16:00:39.625362 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.625078    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t" event={"ID":"9c46e7b8-dbbe-45ec-820e-fea6e25250ef","Type":"ContainerDied","Data":"f95d8a8d0ad6d6007d0700a3c2b37ea932e21c135ab23daccc2019deb466daaa"}
May 20 16:00:39.625362 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.625093    2556 scope.go:117] "RemoveContainer" containerID="c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83"
May 20 16:00:39.631782 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.631762    2556 scope.go:117] "RemoveContainer" containerID="8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105"
May 20 16:00:39.639531 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.639515    2556 scope.go:117] "RemoveContainer" containerID="d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767"
May 20 16:00:39.646459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.646442    2556 scope.go:117] "RemoveContainer" containerID="c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83"
May 20 16:00:39.646716 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:00:39.646697    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83\": container with ID starting with c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83 not found: ID does not exist" containerID="c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83"
May 20 16:00:39.646771 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.646731    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83"} err="failed to get container status \"c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83\": rpc error: code = NotFound desc = could not find container \"c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83\": container with ID starting with c1228d1d6f1c3490f0c4fa36044deea664412371e16a2898ab24e4cb22cd8e83 not found: ID does not exist"
May 20 16:00:39.646771 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.646749    2556 scope.go:117] "RemoveContainer" containerID="8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105"
May 20 16:00:39.646987 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:00:39.646970    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105\": container with ID starting with 8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105 not found: ID does not exist" containerID="8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105"
May 20 16:00:39.647027 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.646994    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105"} err="failed to get container status \"8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105\": rpc error: code = NotFound desc = could not find container \"8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105\": container with ID starting with 8738c35cbac4388999a8123751a85608afa9a196e1ad933ecb316609f5151105 not found: ID does not exist"
May 20 16:00:39.647027 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.647010    2556 scope.go:117] "RemoveContainer" containerID="d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767"
May 20 16:00:39.647280 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:00:39.647252    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767\": container with ID starting with d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767 not found: ID does not exist" containerID="d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767"
May 20 16:00:39.647280 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.647271    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767"} err="failed to get container status \"d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767\": rpc error: code = NotFound desc = could not find container \"d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767\": container with ID starting with d14ad5d8281c9bce875ea77a4bfe6fc717927e4e457294ed1a7ec16ba430b767 not found: ID does not exist"
May 20 16:00:39.687439 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.687396    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79vt7\" (UniqueName: \"kubernetes.io/projected/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kube-api-access-79vt7\") pod \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") "
May 20 16:00:39.687503 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.687438    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") "
May 20 16:00:39.687503 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.687494    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls\") pod \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") "
May 20 16:00:39.687576 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.687512    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kserve-provision-location\") pod \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\" (UID: \"9c46e7b8-dbbe-45ec-820e-fea6e25250ef\") "
May 20 16:00:39.687823 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.687799    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "9c46e7b8-dbbe-45ec-820e-fea6e25250ef" (UID: "9c46e7b8-dbbe-45ec-820e-fea6e25250ef"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:00:39.687951 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.687884    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c46e7b8-dbbe-45ec-820e-fea6e25250ef" (UID: "9c46e7b8-dbbe-45ec-820e-fea6e25250ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:00:39.689451 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.689423    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c46e7b8-dbbe-45ec-820e-fea6e25250ef" (UID: "9c46e7b8-dbbe-45ec-820e-fea6e25250ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:00:39.689518 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.689437    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kube-api-access-79vt7" (OuterVolumeSpecName: "kube-api-access-79vt7") pod "9c46e7b8-dbbe-45ec-820e-fea6e25250ef" (UID: "9c46e7b8-dbbe-45ec-820e-fea6e25250ef"). InnerVolumeSpecName "kube-api-access-79vt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:00:39.788238 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.788213    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:00:39.788238 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.788237    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:00:39.788362 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.788247    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79vt7\" (UniqueName: \"kubernetes.io/projected/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-kube-api-access-79vt7\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:00:39.788362 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.788258    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c46e7b8-dbbe-45ec-820e-fea6e25250ef-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:00:39.942701 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.942652    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"]
May 20 16:00:39.951029 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:39.951007    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-mvg7t"]
May 20 16:00:41.631092 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:41.631061    2556 generic.go:358] "Generic (PLEG): container finished" podID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerID="5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc" exitCode=0
May 20 16:00:41.631441 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:41.631134    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" event={"ID":"10881fdf-ac40-4d21-b924-0a4cc1960dcf","Type":"ContainerDied","Data":"5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc"}
May 20 16:00:41.890931 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:41.890897    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" path="/var/lib/kubelet/pods/9c46e7b8-dbbe-45ec-820e-fea6e25250ef/volumes"
May 20 16:00:42.635227 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:42.635193    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" event={"ID":"10881fdf-ac40-4d21-b924-0a4cc1960dcf","Type":"ContainerStarted","Data":"d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e"}
May 20 16:00:42.635227 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:42.635233    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" event={"ID":"10881fdf-ac40-4d21-b924-0a4cc1960dcf","Type":"ContainerStarted","Data":"3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9"}
May 20 16:00:42.635637 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:42.635428    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:42.654283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:42.654242    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podStartSLOduration=6.6542304439999995 podStartE2EDuration="6.654230444s" podCreationTimestamp="2026-05-20 16:00:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:00:42.652960863 +0000 UTC m=+1685.344216791" watchObservedRunningTime="2026-05-20 16:00:42.654230444 +0000 UTC m=+1685.345486372"
May 20 16:00:43.637759 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:43.637728    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:43.639155 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:43.639125    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:00:44.640358 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:44.640313    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:00:49.644292 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:49.644266    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:00:49.644737 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:49.644712    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:00:59.645006 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:00:59.644926    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:01:09.644774 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:01:09.644730    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:01:19.645181 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:01:19.645144    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:01:29.645121 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:01:29.645067    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:01:39.645677 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:01:39.645629    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:01:49.645218 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:01:49.645179    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:01:59.646064 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:01:59.646038    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:02:07.474518 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.474488    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"]
May 20 16:02:07.474918 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.474786    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" containerID="cri-o://3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9" gracePeriod=30
May 20 16:02:07.474918 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.474835    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kube-rbac-proxy" containerID="cri-o://d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e" gracePeriod=30
May 20 16:02:07.578965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.578938    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"]
May 20 16:02:07.579229 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.579216    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="storage-initializer"
May 20 16:02:07.579281 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.579231    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="storage-initializer"
May 20 16:02:07.579281 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.579240    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kube-rbac-proxy"
May 20 16:02:07.579281 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.579245    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kube-rbac-proxy"
May 20 16:02:07.579281 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.579254    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container"
May 20 16:02:07.579281 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.579260    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container"
May 20 16:02:07.579434 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.579318    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kserve-container"
May 20 16:02:07.579434 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.579327    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c46e7b8-dbbe-45ec-820e-fea6e25250ef" containerName="kube-rbac-proxy"
May 20 16:02:07.583279 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.583262    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.585613 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.585593    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
May 20 16:02:07.585698 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.585614    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
May 20 16:02:07.591385 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.591366    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"]
May 20 16:02:07.695573 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.695546    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e189590-116f-4318-86b6-c4ca0b80b873-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.695681 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.695578    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjgtp\" (UniqueName: \"kubernetes.io/projected/6e189590-116f-4318-86b6-c4ca0b80b873-kube-api-access-rjgtp\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.695681 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.695602    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e189590-116f-4318-86b6-c4ca0b80b873-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.695681 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.695662    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e189590-116f-4318-86b6-c4ca0b80b873-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.796566 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.796543    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e189590-116f-4318-86b6-c4ca0b80b873-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.796682 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.796592    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e189590-116f-4318-86b6-c4ca0b80b873-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.796682 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.796613    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rjgtp\" (UniqueName: \"kubernetes.io/projected/6e189590-116f-4318-86b6-c4ca0b80b873-kube-api-access-rjgtp\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.796682 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.796641    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e189590-116f-4318-86b6-c4ca0b80b873-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.797038 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.797020    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e189590-116f-4318-86b6-c4ca0b80b873-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.797194 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.797170    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e189590-116f-4318-86b6-c4ca0b80b873-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.798884 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.798864    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e189590-116f-4318-86b6-c4ca0b80b873-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.805671 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.805652    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjgtp\" (UniqueName: \"kubernetes.io/projected/6e189590-116f-4318-86b6-c4ca0b80b873-kube-api-access-rjgtp\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:07.876041 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.876017    2556 generic.go:358] "Generic (PLEG): container finished" podID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerID="d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e" exitCode=2
May 20 16:02:07.876153 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.876075    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" event={"ID":"10881fdf-ac40-4d21-b924-0a4cc1960dcf","Type":"ContainerDied","Data":"d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e"}
May 20 16:02:07.893452 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:07.893436    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:08.013362 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:08.013336    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"]
May 20 16:02:08.015314 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:02:08.015289    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e189590_116f_4318_86b6_c4ca0b80b873.slice/crio-46219bd194e78a81fa93ecb792d1532fbd5570236cabab5997b39154fc22c10d WatchSource:0}: Error finding container 46219bd194e78a81fa93ecb792d1532fbd5570236cabab5997b39154fc22c10d: Status 404 returned error can't find the container with id 46219bd194e78a81fa93ecb792d1532fbd5570236cabab5997b39154fc22c10d
May 20 16:02:08.880018 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:08.879989    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" event={"ID":"6e189590-116f-4318-86b6-c4ca0b80b873","Type":"ContainerStarted","Data":"b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b"}
May 20 16:02:08.880400 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:08.880024    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" event={"ID":"6e189590-116f-4318-86b6-c4ca0b80b873","Type":"ContainerStarted","Data":"46219bd194e78a81fa93ecb792d1532fbd5570236cabab5997b39154fc22c10d"}
May 20 16:02:09.640835 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:09.640792    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
May 20 16:02:09.645168 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:09.645137    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 20 16:02:10.813814 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.813791    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:02:10.887301 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.887231    2556 generic.go:358] "Generic (PLEG): container finished" podID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerID="3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9" exitCode=0
May 20 16:02:10.887301 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.887290    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" event={"ID":"10881fdf-ac40-4d21-b924-0a4cc1960dcf","Type":"ContainerDied","Data":"3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9"}
May 20 16:02:10.887460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.887310    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"
May 20 16:02:10.887460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.887320    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt" event={"ID":"10881fdf-ac40-4d21-b924-0a4cc1960dcf","Type":"ContainerDied","Data":"33e07bc1958abf5f3d7064777e9a0ccdc245fe694f05d55fe5610be3ae0815de"}
May 20 16:02:10.887460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.887340    2556 scope.go:117] "RemoveContainer" containerID="d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e"
May 20 16:02:10.894730 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.894713    2556 scope.go:117] "RemoveContainer" containerID="3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9"
May 20 16:02:10.901127 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.901112    2556 scope.go:117] "RemoveContainer" containerID="5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc"
May 20 16:02:10.907174 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.907160    2556 scope.go:117] "RemoveContainer" containerID="d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e"
May 20 16:02:10.907402 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:02:10.907385    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e\": container with ID starting with d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e not found: ID does not exist" containerID="d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e"
May 20 16:02:10.907452 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.907409    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e"} err="failed to get container status \"d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e\": rpc error: code = NotFound desc = could not find container \"d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e\": container with ID starting with d8c940a9d6798aba99b3201851ae4a3b97341985e1883c1f7db6b9b7b8ad788e not found: ID does not exist"
May 20 16:02:10.907452 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.907425    2556 scope.go:117] "RemoveContainer" containerID="3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9"
May 20 16:02:10.907620 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:02:10.907606    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9\": container with ID starting with 3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9 not found: ID does not exist" containerID="3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9"
May 20 16:02:10.907665 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.907623    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9"} err="failed to get container status \"3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9\": rpc error: code = NotFound desc = could not find container \"3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9\": container with ID starting with 3a5f87874f044d7e7bc6a3f7e24293fb2ca6815d5fe29127deaf852301e6beb9 not found: ID does not exist"
May 20 16:02:10.907665 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.907635    2556 scope.go:117] "RemoveContainer" containerID="5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc"
May 20 16:02:10.907832 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:02:10.907813    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc\": container with ID starting with 5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc not found: ID does not exist" containerID="5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc"
May 20 16:02:10.907871 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.907838    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc"} err="failed to get container status \"5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc\": rpc error: code = NotFound desc = could not find container \"5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc\": container with ID starting with 5827820f9195c9428c71efdf4631af318852c15e45dec9a8be1522c6fb32a8cc not found: ID does not exist"
May 20 16:02:10.917079 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.917060    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64krj\" (UniqueName: \"kubernetes.io/projected/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kube-api-access-64krj\") pod \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") "
May 20 16:02:10.917163 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.917091    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls\") pod \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") "
May 20 16:02:10.917163 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.917153    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kserve-provision-location\") pod \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") "
May 20 16:02:10.917234 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.917187    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10881fdf-ac40-4d21-b924-0a4cc1960dcf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\" (UID: \"10881fdf-ac40-4d21-b924-0a4cc1960dcf\") "
May 20 16:02:10.917488 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.917466    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10881fdf-ac40-4d21-b924-0a4cc1960dcf" (UID: "10881fdf-ac40-4d21-b924-0a4cc1960dcf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:02:10.917579 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.917557    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10881fdf-ac40-4d21-b924-0a4cc1960dcf-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "10881fdf-ac40-4d21-b924-0a4cc1960dcf" (UID: "10881fdf-ac40-4d21-b924-0a4cc1960dcf"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:02:10.918994 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.918974    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kube-api-access-64krj" (OuterVolumeSpecName: "kube-api-access-64krj") pod "10881fdf-ac40-4d21-b924-0a4cc1960dcf" (UID: "10881fdf-ac40-4d21-b924-0a4cc1960dcf"). InnerVolumeSpecName "kube-api-access-64krj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:02:10.919067 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:10.919047    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10881fdf-ac40-4d21-b924-0a4cc1960dcf" (UID: "10881fdf-ac40-4d21-b924-0a4cc1960dcf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:02:11.018542 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:11.018521    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64krj\" (UniqueName: \"kubernetes.io/projected/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kube-api-access-64krj\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:02:11.018542 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:11.018540    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10881fdf-ac40-4d21-b924-0a4cc1960dcf-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:02:11.018654 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:11.018551    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10881fdf-ac40-4d21-b924-0a4cc1960dcf-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:02:11.018654 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:11.018560    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10881fdf-ac40-4d21-b924-0a4cc1960dcf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:02:11.211167 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:11.211139    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"]
May 20 16:02:11.217509 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:11.217487    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-7qfgt"]
May 20 16:02:11.892018 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:11.891971    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" path="/var/lib/kubelet/pods/10881fdf-ac40-4d21-b924-0a4cc1960dcf/volumes"
May 20 16:02:12.896366 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:12.896331    2556 generic.go:358] "Generic (PLEG): container finished" podID="6e189590-116f-4318-86b6-c4ca0b80b873" containerID="b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b" exitCode=0
May 20 16:02:12.896838 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:12.896413    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" event={"ID":"6e189590-116f-4318-86b6-c4ca0b80b873","Type":"ContainerDied","Data":"b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b"}
May 20 16:02:13.900392 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:13.900351    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" event={"ID":"6e189590-116f-4318-86b6-c4ca0b80b873","Type":"ContainerStarted","Data":"fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045"}
May 20 16:02:13.900392 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:13.900386    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" event={"ID":"6e189590-116f-4318-86b6-c4ca0b80b873","Type":"ContainerStarted","Data":"566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725"}
May 20 16:02:13.900850 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:13.900656    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:13.900850 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:13.900792    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:13.901823 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:13.901797    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:02:13.926055 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:13.926004    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podStartSLOduration=6.925992165 podStartE2EDuration="6.925992165s" podCreationTimestamp="2026-05-20 16:02:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:02:13.924581255 +0000 UTC m=+1776.615837183" watchObservedRunningTime="2026-05-20 16:02:13.925992165 +0000 UTC m=+1776.617248093"
May 20 16:02:14.903217 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:14.903176    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:02:15.905558 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:15.905514    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:02:20.910061 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:20.910018    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:02:20.910655 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:20.910596    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:02:30.910633 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:30.910532    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:02:40.910601 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:40.910568    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:02:50.911523 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:02:50.911476    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:03:00.911432 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:00.911384    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:03:10.910782 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:10.910746    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:03:20.910796 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:20.910753    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:03:30.911759 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:30.911724    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:03:38.595150 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.595115    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"]
May 20 16:03:38.595553 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.595438    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" containerID="cri-o://566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725" gracePeriod=30
May 20 16:03:38.595553 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.595499    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kube-rbac-proxy" containerID="cri-o://fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045" gracePeriod=30
May 20 16:03:38.691950 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.691924    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"]
May 20 16:03:38.692251 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.692235    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container"
May 20 16:03:38.692339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.692254    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container"
May 20 16:03:38.692339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.692280    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="storage-initializer"
May 20 16:03:38.692339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.692290    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="storage-initializer"
May 20 16:03:38.692339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.692299    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kube-rbac-proxy"
May 20 16:03:38.692339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.692308    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kube-rbac-proxy"
May 20 16:03:38.692605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.692369    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kserve-container"
May 20 16:03:38.692605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.692380    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="10881fdf-ac40-4d21-b924-0a4cc1960dcf" containerName="kube-rbac-proxy"
May 20 16:03:38.695020 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.695004    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.697184 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.697162    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a8175b-predictor-serving-cert\""
May 20 16:03:38.697295 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.697269    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a8175b-kube-rbac-proxy-sar-config\""
May 20 16:03:38.703680 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.703658    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"]
May 20 16:03:38.730420 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.730379    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb987d2e-a0ae-400d-88d8-637a6b681b35-proxy-tls\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.730522 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.730445    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb987d2e-a0ae-400d-88d8-637a6b681b35-isvc-primary-a8175b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.730522 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.730505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bdhg\" (UniqueName: \"kubernetes.io/projected/bb987d2e-a0ae-400d-88d8-637a6b681b35-kube-api-access-2bdhg\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.730667 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.730530    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb987d2e-a0ae-400d-88d8-637a6b681b35-kserve-provision-location\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.830783 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.830758    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb987d2e-a0ae-400d-88d8-637a6b681b35-proxy-tls\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.830878 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.830798    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb987d2e-a0ae-400d-88d8-637a6b681b35-isvc-primary-a8175b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.830878 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.830831    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bdhg\" (UniqueName: \"kubernetes.io/projected/bb987d2e-a0ae-400d-88d8-637a6b681b35-kube-api-access-2bdhg\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.830878 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.830847    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb987d2e-a0ae-400d-88d8-637a6b681b35-kserve-provision-location\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.831225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.831206    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb987d2e-a0ae-400d-88d8-637a6b681b35-kserve-provision-location\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.831499 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.831478    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb987d2e-a0ae-400d-88d8-637a6b681b35-isvc-primary-a8175b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.833050 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.833030    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb987d2e-a0ae-400d-88d8-637a6b681b35-proxy-tls\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:38.840789 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:38.840771    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bdhg\" (UniqueName: \"kubernetes.io/projected/bb987d2e-a0ae-400d-88d8-637a6b681b35-kube-api-access-2bdhg\") pod \"isvc-primary-a8175b-predictor-5fb455746d-2hm67\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") " pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:39.006059 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:39.006027    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:39.121044 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:39.121016    2556 generic.go:358] "Generic (PLEG): container finished" podID="6e189590-116f-4318-86b6-c4ca0b80b873" containerID="fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045" exitCode=2
May 20 16:03:39.121187 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:39.121088    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" event={"ID":"6e189590-116f-4318-86b6-c4ca0b80b873","Type":"ContainerDied","Data":"fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045"}
May 20 16:03:39.132135 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:39.132092    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"]
May 20 16:03:39.134984 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:03:39.134960    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb987d2e_a0ae_400d_88d8_637a6b681b35.slice/crio-9c34dac30b498900a1879a5f4e4aba40161fa15f467ca4d12d3001081f5434bd WatchSource:0}: Error finding container 9c34dac30b498900a1879a5f4e4aba40161fa15f467ca4d12d3001081f5434bd: Status 404 returned error can't find the container with id 9c34dac30b498900a1879a5f4e4aba40161fa15f467ca4d12d3001081f5434bd
May 20 16:03:39.136614 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:39.136599    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:03:40.125193 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:40.125159    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" event={"ID":"bb987d2e-a0ae-400d-88d8-637a6b681b35","Type":"ContainerStarted","Data":"b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe"}
May 20 16:03:40.125193 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:40.125199    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" event={"ID":"bb987d2e-a0ae-400d-88d8-637a6b681b35","Type":"ContainerStarted","Data":"9c34dac30b498900a1879a5f4e4aba40161fa15f467ca4d12d3001081f5434bd"}
May 20 16:03:40.906187 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:40.906153    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
May 20 16:03:40.910596 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:40.910571    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 20 16:03:41.735818 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.735796    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:03:41.749952 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.749928    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e189590-116f-4318-86b6-c4ca0b80b873-kserve-provision-location\") pod \"6e189590-116f-4318-86b6-c4ca0b80b873\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") "
May 20 16:03:41.750031 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.749971    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e189590-116f-4318-86b6-c4ca0b80b873-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"6e189590-116f-4318-86b6-c4ca0b80b873\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") "
May 20 16:03:41.750031 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.749998    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e189590-116f-4318-86b6-c4ca0b80b873-proxy-tls\") pod \"6e189590-116f-4318-86b6-c4ca0b80b873\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") "
May 20 16:03:41.750139 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.750068    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjgtp\" (UniqueName: \"kubernetes.io/projected/6e189590-116f-4318-86b6-c4ca0b80b873-kube-api-access-rjgtp\") pod \"6e189590-116f-4318-86b6-c4ca0b80b873\" (UID: \"6e189590-116f-4318-86b6-c4ca0b80b873\") "
May 20 16:03:41.750235 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.750214    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e189590-116f-4318-86b6-c4ca0b80b873-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e189590-116f-4318-86b6-c4ca0b80b873" (UID: "6e189590-116f-4318-86b6-c4ca0b80b873"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:41.750300 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.750275    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e189590-116f-4318-86b6-c4ca0b80b873-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:03:41.750360 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.750316    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e189590-116f-4318-86b6-c4ca0b80b873-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "6e189590-116f-4318-86b6-c4ca0b80b873" (UID: "6e189590-116f-4318-86b6-c4ca0b80b873"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:03:41.751923 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.751896    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e189590-116f-4318-86b6-c4ca0b80b873-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e189590-116f-4318-86b6-c4ca0b80b873" (UID: "6e189590-116f-4318-86b6-c4ca0b80b873"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:03:41.752036 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.752009    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e189590-116f-4318-86b6-c4ca0b80b873-kube-api-access-rjgtp" (OuterVolumeSpecName: "kube-api-access-rjgtp") pod "6e189590-116f-4318-86b6-c4ca0b80b873" (UID: "6e189590-116f-4318-86b6-c4ca0b80b873"). InnerVolumeSpecName "kube-api-access-rjgtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:03:41.850964 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.850909    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjgtp\" (UniqueName: \"kubernetes.io/projected/6e189590-116f-4318-86b6-c4ca0b80b873-kube-api-access-rjgtp\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:03:41.850964 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.850930    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e189590-116f-4318-86b6-c4ca0b80b873-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:03:41.850964 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:41.850941    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e189590-116f-4318-86b6-c4ca0b80b873-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:03:42.132681 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.132615    2556 generic.go:358] "Generic (PLEG): container finished" podID="6e189590-116f-4318-86b6-c4ca0b80b873" containerID="566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725" exitCode=0
May 20 16:03:42.132793 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.132697    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"
May 20 16:03:42.132793 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.132700    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" event={"ID":"6e189590-116f-4318-86b6-c4ca0b80b873","Type":"ContainerDied","Data":"566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725"}
May 20 16:03:42.132793 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.132742    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4" event={"ID":"6e189590-116f-4318-86b6-c4ca0b80b873","Type":"ContainerDied","Data":"46219bd194e78a81fa93ecb792d1532fbd5570236cabab5997b39154fc22c10d"}
May 20 16:03:42.132793 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.132763    2556 scope.go:117] "RemoveContainer" containerID="fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045"
May 20 16:03:42.140826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.140805    2556 scope.go:117] "RemoveContainer" containerID="566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725"
May 20 16:03:42.148132 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.148116    2556 scope.go:117] "RemoveContainer" containerID="b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b"
May 20 16:03:42.153733 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.153710    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"]
May 20 16:03:42.155791 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.155777    2556 scope.go:117] "RemoveContainer" containerID="fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045"
May 20 16:03:42.156054 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:03:42.156036    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045\": container with ID starting with fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045 not found: ID does not exist" containerID="fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045"
May 20 16:03:42.156123 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.156061    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045"} err="failed to get container status \"fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045\": rpc error: code = NotFound desc = could not find container \"fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045\": container with ID starting with fa2267a0ac4742f990875e7b096e5a9a6780ae2d8527996ebbac25a26ec9a045 not found: ID does not exist"
May 20 16:03:42.156123 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.156078    2556 scope.go:117] "RemoveContainer" containerID="566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725"
May 20 16:03:42.156303 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:03:42.156286    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725\": container with ID starting with 566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725 not found: ID does not exist" containerID="566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725"
May 20 16:03:42.156342 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.156307    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725"} err="failed to get container status \"566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725\": rpc error: code = NotFound desc = could not find container \"566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725\": container with ID starting with 566b289b62fd9adcf708cf7c9055deaebd2f481aa463c61a036c3366d710c725 not found: ID does not exist"
May 20 16:03:42.156342 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.156320    2556 scope.go:117] "RemoveContainer" containerID="b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b"
May 20 16:03:42.156540 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:03:42.156518    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b\": container with ID starting with b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b not found: ID does not exist" containerID="b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b"
May 20 16:03:42.156583 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.156547    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b"} err="failed to get container status \"b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b\": rpc error: code = NotFound desc = could not find container \"b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b\": container with ID starting with b900e743f24c2003debe9a0508193d227c55b832cdbdc692cc6444a2cacb491b not found: ID does not exist"
May 20 16:03:42.161216 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:42.161182    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-mzmw4"]
May 20 16:03:43.137150 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:43.137055    2556 generic.go:358] "Generic (PLEG): container finished" podID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerID="b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe" exitCode=0
May 20 16:03:43.137551 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:43.137137    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" event={"ID":"bb987d2e-a0ae-400d-88d8-637a6b681b35","Type":"ContainerDied","Data":"b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe"}
May 20 16:03:43.891252 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:43.891219    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" path="/var/lib/kubelet/pods/6e189590-116f-4318-86b6-c4ca0b80b873/volumes"
May 20 16:03:44.142397 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:44.142316    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" event={"ID":"bb987d2e-a0ae-400d-88d8-637a6b681b35","Type":"ContainerStarted","Data":"7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390"}
May 20 16:03:44.142397 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:44.142361    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" event={"ID":"bb987d2e-a0ae-400d-88d8-637a6b681b35","Type":"ContainerStarted","Data":"e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44"}
May 20 16:03:44.142788 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:44.142579    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:44.161400 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:44.161360    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podStartSLOduration=6.16134758 podStartE2EDuration="6.16134758s" podCreationTimestamp="2026-05-20 16:03:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:03:44.15967067 +0000 UTC m=+1866.850926599" watchObservedRunningTime="2026-05-20 16:03:44.16134758 +0000 UTC m=+1866.852603508"
May 20 16:03:45.145468 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:45.145436    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:45.146616 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:45.146592    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 20 16:03:46.148182 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:46.148127    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 20 16:03:51.152758 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:51.152728    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:03:51.153200 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:03:51.153171    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 20 16:04:01.153957 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:01.153862    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 20 16:04:11.153277 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:11.153236    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 20 16:04:21.154056 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:21.154015    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 20 16:04:31.153256 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:31.153213    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 20 16:04:41.153557 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:41.153510    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
May 20 16:04:51.154325 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:51.154287    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:04:58.872199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872162    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"]
May 20 16:04:58.872592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872443    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container"
May 20 16:04:58.872592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872454    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container"
May 20 16:04:58.872592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872471    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="storage-initializer"
May 20 16:04:58.872592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872477    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="storage-initializer"
May 20 16:04:58.872592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872483    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kube-rbac-proxy"
May 20 16:04:58.872592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872489    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kube-rbac-proxy"
May 20 16:04:58.872592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872532    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kserve-container"
May 20 16:04:58.872592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.872540    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="6e189590-116f-4318-86b6-c4ca0b80b873" containerName="kube-rbac-proxy"
May 20 16:04:58.874500 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.874480    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:58.876981 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.876953    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a8175b-predictor-serving-cert\""
May 20 16:04:58.876981 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.876977    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a8175b-kube-rbac-proxy-sar-config\""
May 20 16:04:58.877240 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.877047    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 16:04:58.877240 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.877048    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a8175b-dockercfg-d87sc\""
May 20 16:04:58.877955 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.877934    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a8175b\""
May 20 16:04:58.885797 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.885774    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"]
May 20 16:04:58.981964 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.981938    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-cabundle-cert\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:58.982086 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.981980    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae92600-44fb-4ba8-aa45-55bec06cd852-proxy-tls\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:58.982086 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.982006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae92600-44fb-4ba8-aa45-55bec06cd852-kserve-provision-location\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:58.982252 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.982134    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv2bc\" (UniqueName: \"kubernetes.io/projected/9ae92600-44fb-4ba8-aa45-55bec06cd852-kube-api-access-sv2bc\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:58.982252 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:58.982221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-isvc-secondary-a8175b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.083570 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.083541    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-cabundle-cert\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.083705 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.083573    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae92600-44fb-4ba8-aa45-55bec06cd852-proxy-tls\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.083705 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.083592    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae92600-44fb-4ba8-aa45-55bec06cd852-kserve-provision-location\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.083705 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.083611    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sv2bc\" (UniqueName: \"kubernetes.io/projected/9ae92600-44fb-4ba8-aa45-55bec06cd852-kube-api-access-sv2bc\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.083705 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.083648    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-isvc-secondary-a8175b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.084016 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.083995    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae92600-44fb-4ba8-aa45-55bec06cd852-kserve-provision-location\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.084354 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.084329    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-isvc-secondary-a8175b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.084445 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.084351    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-cabundle-cert\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.086277 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.086256    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae92600-44fb-4ba8-aa45-55bec06cd852-proxy-tls\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.092185 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.092162    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv2bc\" (UniqueName: \"kubernetes.io/projected/9ae92600-44fb-4ba8-aa45-55bec06cd852-kube-api-access-sv2bc\") pod \"isvc-secondary-a8175b-predictor-c9d486855-sth99\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") " pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.185143 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.185070    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:04:59.315951 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.315929    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"]
May 20 16:04:59.318365 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:04:59.318338    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae92600_44fb_4ba8_aa45_55bec06cd852.slice/crio-c37bc6f43dd49e98a65025d269950d99b934981b1a69a3312e811924e608f0be WatchSource:0}: Error finding container c37bc6f43dd49e98a65025d269950d99b934981b1a69a3312e811924e608f0be: Status 404 returned error can't find the container with id c37bc6f43dd49e98a65025d269950d99b934981b1a69a3312e811924e608f0be
May 20 16:04:59.344649 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:04:59.344626    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99" event={"ID":"9ae92600-44fb-4ba8-aa45-55bec06cd852","Type":"ContainerStarted","Data":"c37bc6f43dd49e98a65025d269950d99b934981b1a69a3312e811924e608f0be"}
May 20 16:05:00.348330 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:00.348289    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99" event={"ID":"9ae92600-44fb-4ba8-aa45-55bec06cd852","Type":"ContainerStarted","Data":"561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917"}
May 20 16:05:04.358888 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:04.358861    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a8175b-predictor-c9d486855-sth99_9ae92600-44fb-4ba8-aa45-55bec06cd852/storage-initializer/0.log"
May 20 16:05:04.359283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:04.358899    2556 generic.go:358] "Generic (PLEG): container finished" podID="9ae92600-44fb-4ba8-aa45-55bec06cd852" containerID="561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917" exitCode=1
May 20 16:05:04.359283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:04.358934    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99" event={"ID":"9ae92600-44fb-4ba8-aa45-55bec06cd852","Type":"ContainerDied","Data":"561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917"}
May 20 16:05:05.363390 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:05.363365    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a8175b-predictor-c9d486855-sth99_9ae92600-44fb-4ba8-aa45-55bec06cd852/storage-initializer/0.log"
May 20 16:05:05.363788 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:05.363455    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99" event={"ID":"9ae92600-44fb-4ba8-aa45-55bec06cd852","Type":"ContainerStarted","Data":"629cd7db275814e6491935b7ece42209f2097fde382384c25c7b7d9bbc8850fa"}
May 20 16:05:09.377807 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:09.377781    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a8175b-predictor-c9d486855-sth99_9ae92600-44fb-4ba8-aa45-55bec06cd852/storage-initializer/1.log"
May 20 16:05:09.378312 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:09.378159    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a8175b-predictor-c9d486855-sth99_9ae92600-44fb-4ba8-aa45-55bec06cd852/storage-initializer/0.log"
May 20 16:05:09.378312 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:09.378204    2556 generic.go:358] "Generic (PLEG): container finished" podID="9ae92600-44fb-4ba8-aa45-55bec06cd852" containerID="629cd7db275814e6491935b7ece42209f2097fde382384c25c7b7d9bbc8850fa" exitCode=1
May 20 16:05:09.378312 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:09.378245    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99" event={"ID":"9ae92600-44fb-4ba8-aa45-55bec06cd852","Type":"ContainerDied","Data":"629cd7db275814e6491935b7ece42209f2097fde382384c25c7b7d9bbc8850fa"}
May 20 16:05:09.378312 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:09.378285    2556 scope.go:117] "RemoveContainer" containerID="561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917"
May 20 16:05:09.378646 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:09.378626    2556 scope.go:117] "RemoveContainer" containerID="561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917"
May 20 16:05:09.387633 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:05:09.387607    2556 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a8175b-predictor-c9d486855-sth99_kserve-ci-e2e-test_9ae92600-44fb-4ba8-aa45-55bec06cd852_0 in pod sandbox c37bc6f43dd49e98a65025d269950d99b934981b1a69a3312e811924e608f0be from index: no such id: '561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917'" containerID="561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917"
May 20 16:05:09.387716 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:09.387647    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a8175b-predictor-c9d486855-sth99_kserve-ci-e2e-test_9ae92600-44fb-4ba8-aa45-55bec06cd852_0 in pod sandbox c37bc6f43dd49e98a65025d269950d99b934981b1a69a3312e811924e608f0be from index: no such id: '561372b5522f0296dbbc55dd9020c477fb72624766813e458b4146d99a7a8917'"
May 20 16:05:09.387767 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:05:09.387750    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a8175b-predictor-c9d486855-sth99_kserve-ci-e2e-test(9ae92600-44fb-4ba8-aa45-55bec06cd852)\"" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99" podUID="9ae92600-44fb-4ba8-aa45-55bec06cd852"
May 20 16:05:10.382337 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:10.382313    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a8175b-predictor-c9d486855-sth99_9ae92600-44fb-4ba8-aa45-55bec06cd852/storage-initializer/1.log"
May 20 16:05:14.911032 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:14.910992    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"]
May 20 16:05:14.987067 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:14.987034    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"]
May 20 16:05:14.987410 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:14.987377    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container" containerID="cri-o://e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44" gracePeriod=30
May 20 16:05:14.987690 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:14.987659    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kube-rbac-proxy" containerID="cri-o://7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390" gracePeriod=30
May 20 16:05:15.043017 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.042985    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"]
May 20 16:05:15.045243 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.045227    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.047763 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.047747    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-3cec1b-predictor-serving-cert\""
May 20 16:05:15.047869 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.047811    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\""
May 20 16:05:15.047974 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.047961    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-3cec1b-dockercfg-qbnhf\""
May 20 16:05:15.048032 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.048019    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-3cec1b\""
May 20 16:05:15.057631 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.057613    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"]
May 20 16:05:15.105343 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.105327    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a8175b-predictor-c9d486855-sth99_9ae92600-44fb-4ba8-aa45-55bec06cd852/storage-initializer/1.log"
May 20 16:05:15.105433 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.105384    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:05:15.196515 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196465    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae92600-44fb-4ba8-aa45-55bec06cd852-kserve-provision-location\") pod \"9ae92600-44fb-4ba8-aa45-55bec06cd852\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") "
May 20 16:05:15.196515 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196501    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae92600-44fb-4ba8-aa45-55bec06cd852-proxy-tls\") pod \"9ae92600-44fb-4ba8-aa45-55bec06cd852\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") "
May 20 16:05:15.196658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196536    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-isvc-secondary-a8175b-kube-rbac-proxy-sar-config\") pod \"9ae92600-44fb-4ba8-aa45-55bec06cd852\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") "
May 20 16:05:15.196658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196562    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-cabundle-cert\") pod \"9ae92600-44fb-4ba8-aa45-55bec06cd852\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") "
May 20 16:05:15.196658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196592    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv2bc\" (UniqueName: \"kubernetes.io/projected/9ae92600-44fb-4ba8-aa45-55bec06cd852-kube-api-access-sv2bc\") pod \"9ae92600-44fb-4ba8-aa45-55bec06cd852\" (UID: \"9ae92600-44fb-4ba8-aa45-55bec06cd852\") "
May 20 16:05:15.196826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196689    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhxc\" (UniqueName: \"kubernetes.io/projected/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kube-api-access-rmhxc\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.196826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196712    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae92600-44fb-4ba8-aa45-55bec06cd852-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ae92600-44fb-4ba8-aa45-55bec06cd852" (UID: "9ae92600-44fb-4ba8-aa45-55bec06cd852"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:15.196826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196740    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.196826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196764    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020aa1f2-0e06-4309-a1b4-39ad7beddee0-proxy-tls\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.196826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-cabundle-cert\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.196826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kserve-provision-location\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.197066 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196863    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ae92600-44fb-4ba8-aa45-55bec06cd852-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:15.197066 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196905    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ae92600-44fb-4ba8-aa45-55bec06cd852" (UID: "9ae92600-44fb-4ba8-aa45-55bec06cd852"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:05:15.197066 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.196915    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-isvc-secondary-a8175b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-a8175b-kube-rbac-proxy-sar-config") pod "9ae92600-44fb-4ba8-aa45-55bec06cd852" (UID: "9ae92600-44fb-4ba8-aa45-55bec06cd852"). InnerVolumeSpecName "isvc-secondary-a8175b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:05:15.198537 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.198519    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae92600-44fb-4ba8-aa45-55bec06cd852-kube-api-access-sv2bc" (OuterVolumeSpecName: "kube-api-access-sv2bc") pod "9ae92600-44fb-4ba8-aa45-55bec06cd852" (UID: "9ae92600-44fb-4ba8-aa45-55bec06cd852"). InnerVolumeSpecName "kube-api-access-sv2bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:05:15.198607 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.198552    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae92600-44fb-4ba8-aa45-55bec06cd852-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ae92600-44fb-4ba8-aa45-55bec06cd852" (UID: "9ae92600-44fb-4ba8-aa45-55bec06cd852"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:05:15.297192 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297166    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhxc\" (UniqueName: \"kubernetes.io/projected/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kube-api-access-rmhxc\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.297277 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297218    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.297277 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297242    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020aa1f2-0e06-4309-a1b4-39ad7beddee0-proxy-tls\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.297277 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297267    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-cabundle-cert\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.297394 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297293    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kserve-provision-location\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.297394 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297341    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae92600-44fb-4ba8-aa45-55bec06cd852-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:15.297394 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297356    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-isvc-secondary-a8175b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:15.297394 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297370    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ae92600-44fb-4ba8-aa45-55bec06cd852-cabundle-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:15.297394 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297384    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv2bc\" (UniqueName: \"kubernetes.io/projected/9ae92600-44fb-4ba8-aa45-55bec06cd852-kube-api-access-sv2bc\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:15.297713 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297695    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kserve-provision-location\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.297981 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.297958    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-cabundle-cert\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.298166 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.298149    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.299445 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.299427    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020aa1f2-0e06-4309-a1b4-39ad7beddee0-proxy-tls\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.305744 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.305726    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhxc\" (UniqueName: \"kubernetes.io/projected/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kube-api-access-rmhxc\") pod \"isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") " pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.354562 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.354534    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:15.400848 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.400816    2556 generic.go:358] "Generic (PLEG): container finished" podID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerID="7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390" exitCode=2
May 20 16:05:15.400974 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.400909    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" event={"ID":"bb987d2e-a0ae-400d-88d8-637a6b681b35","Type":"ContainerDied","Data":"7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390"}
May 20 16:05:15.402215 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.402197    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a8175b-predictor-c9d486855-sth99_9ae92600-44fb-4ba8-aa45-55bec06cd852/storage-initializer/1.log"
May 20 16:05:15.402341 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.402285    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99" event={"ID":"9ae92600-44fb-4ba8-aa45-55bec06cd852","Type":"ContainerDied","Data":"c37bc6f43dd49e98a65025d269950d99b934981b1a69a3312e811924e608f0be"}
May 20 16:05:15.402341 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.402323    2556 scope.go:117] "RemoveContainer" containerID="629cd7db275814e6491935b7ece42209f2097fde382384c25c7b7d9bbc8850fa"
May 20 16:05:15.402462 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.402337    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"
May 20 16:05:15.442834 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.442809    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"]
May 20 16:05:15.445866 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.445844    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a8175b-predictor-c9d486855-sth99"]
May 20 16:05:15.479406 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.479388    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"]
May 20 16:05:15.481813 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:05:15.481787    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020aa1f2_0e06_4309_a1b4_39ad7beddee0.slice/crio-b6875f52d16350a215163cda57713bf44dd641e620062e6decb188f53e3f2587 WatchSource:0}: Error finding container b6875f52d16350a215163cda57713bf44dd641e620062e6decb188f53e3f2587: Status 404 returned error can't find the container with id b6875f52d16350a215163cda57713bf44dd641e620062e6decb188f53e3f2587
May 20 16:05:15.891236 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:15.891209    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae92600-44fb-4ba8-aa45-55bec06cd852" path="/var/lib/kubelet/pods/9ae92600-44fb-4ba8-aa45-55bec06cd852/volumes"
May 20 16:05:16.148905 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:16.148831    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
May 20 16:05:16.407158 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:16.407065    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf" event={"ID":"020aa1f2-0e06-4309-a1b4-39ad7beddee0","Type":"ContainerStarted","Data":"7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590"}
May 20 16:05:16.407158 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:16.407116    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf" event={"ID":"020aa1f2-0e06-4309-a1b4-39ad7beddee0","Type":"ContainerStarted","Data":"b6875f52d16350a215163cda57713bf44dd641e620062e6decb188f53e3f2587"}
May 20 16:05:18.414621 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:18.414540    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf_020aa1f2-0e06-4309-a1b4-39ad7beddee0/storage-initializer/0.log"
May 20 16:05:18.414621 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:18.414578    2556 generic.go:358] "Generic (PLEG): container finished" podID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerID="7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590" exitCode=1
May 20 16:05:18.415023 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:18.414659    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf" event={"ID":"020aa1f2-0e06-4309-a1b4-39ad7beddee0","Type":"ContainerDied","Data":"7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590"}
May 20 16:05:19.024846 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.024822    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:05:19.123991 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.123919    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb987d2e-a0ae-400d-88d8-637a6b681b35-kserve-provision-location\") pod \"bb987d2e-a0ae-400d-88d8-637a6b681b35\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") "
May 20 16:05:19.123991 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.123957    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb987d2e-a0ae-400d-88d8-637a6b681b35-isvc-primary-a8175b-kube-rbac-proxy-sar-config\") pod \"bb987d2e-a0ae-400d-88d8-637a6b681b35\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") "
May 20 16:05:19.124210 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.123995    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bdhg\" (UniqueName: \"kubernetes.io/projected/bb987d2e-a0ae-400d-88d8-637a6b681b35-kube-api-access-2bdhg\") pod \"bb987d2e-a0ae-400d-88d8-637a6b681b35\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") "
May 20 16:05:19.124210 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.124058    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb987d2e-a0ae-400d-88d8-637a6b681b35-proxy-tls\") pod \"bb987d2e-a0ae-400d-88d8-637a6b681b35\" (UID: \"bb987d2e-a0ae-400d-88d8-637a6b681b35\") "
May 20 16:05:19.124316 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.124296    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb987d2e-a0ae-400d-88d8-637a6b681b35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb987d2e-a0ae-400d-88d8-637a6b681b35" (UID: "bb987d2e-a0ae-400d-88d8-637a6b681b35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:19.124429 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.124406    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb987d2e-a0ae-400d-88d8-637a6b681b35-isvc-primary-a8175b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-a8175b-kube-rbac-proxy-sar-config") pod "bb987d2e-a0ae-400d-88d8-637a6b681b35" (UID: "bb987d2e-a0ae-400d-88d8-637a6b681b35"). InnerVolumeSpecName "isvc-primary-a8175b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:05:19.126149 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.126123    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb987d2e-a0ae-400d-88d8-637a6b681b35-kube-api-access-2bdhg" (OuterVolumeSpecName: "kube-api-access-2bdhg") pod "bb987d2e-a0ae-400d-88d8-637a6b681b35" (UID: "bb987d2e-a0ae-400d-88d8-637a6b681b35"). InnerVolumeSpecName "kube-api-access-2bdhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:05:19.126205 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.126169    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb987d2e-a0ae-400d-88d8-637a6b681b35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb987d2e-a0ae-400d-88d8-637a6b681b35" (UID: "bb987d2e-a0ae-400d-88d8-637a6b681b35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:05:19.224884 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.224857    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb987d2e-a0ae-400d-88d8-637a6b681b35-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:19.224884 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.224880    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb987d2e-a0ae-400d-88d8-637a6b681b35-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:19.225035 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.224890    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-a8175b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb987d2e-a0ae-400d-88d8-637a6b681b35-isvc-primary-a8175b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:19.225035 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.224901    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bdhg\" (UniqueName: \"kubernetes.io/projected/bb987d2e-a0ae-400d-88d8-637a6b681b35-kube-api-access-2bdhg\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:19.418828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.418762    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf_020aa1f2-0e06-4309-a1b4-39ad7beddee0/storage-initializer/0.log"
May 20 16:05:19.419225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.418881    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf" event={"ID":"020aa1f2-0e06-4309-a1b4-39ad7beddee0","Type":"ContainerStarted","Data":"47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712"}
May 20 16:05:19.420450 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.420427    2556 generic.go:358] "Generic (PLEG): container finished" podID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerID="e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44" exitCode=0
May 20 16:05:19.420560 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.420480    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" event={"ID":"bb987d2e-a0ae-400d-88d8-637a6b681b35","Type":"ContainerDied","Data":"e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44"}
May 20 16:05:19.420560 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.420491    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"
May 20 16:05:19.420560 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.420510    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67" event={"ID":"bb987d2e-a0ae-400d-88d8-637a6b681b35","Type":"ContainerDied","Data":"9c34dac30b498900a1879a5f4e4aba40161fa15f467ca4d12d3001081f5434bd"}
May 20 16:05:19.420560 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.420531    2556 scope.go:117] "RemoveContainer" containerID="7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390"
May 20 16:05:19.427670 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.427652    2556 scope.go:117] "RemoveContainer" containerID="e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44"
May 20 16:05:19.434407 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.434392    2556 scope.go:117] "RemoveContainer" containerID="b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe"
May 20 16:05:19.441168 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.441150    2556 scope.go:117] "RemoveContainer" containerID="7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390"
May 20 16:05:19.441416 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:05:19.441391    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390\": container with ID starting with 7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390 not found: ID does not exist" containerID="7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390"
May 20 16:05:19.441463 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.441423    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390"} err="failed to get container status \"7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390\": rpc error: code = NotFound desc = could not find container \"7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390\": container with ID starting with 7eb58c8b6021a647d856af2dc143d8a1766d00d5d3fecdf6ea63f212e7832390 not found: ID does not exist"
May 20 16:05:19.441463 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.441441    2556 scope.go:117] "RemoveContainer" containerID="e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44"
May 20 16:05:19.441656 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:05:19.441639    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44\": container with ID starting with e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44 not found: ID does not exist" containerID="e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44"
May 20 16:05:19.441696 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.441662    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44"} err="failed to get container status \"e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44\": rpc error: code = NotFound desc = could not find container \"e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44\": container with ID starting with e0e8bf91c190c81c63b84cbbd50908e78f05607e72787682448628276810ec44 not found: ID does not exist"
May 20 16:05:19.441696 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.441678    2556 scope.go:117] "RemoveContainer" containerID="b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe"
May 20 16:05:19.441896 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:05:19.441882    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe\": container with ID starting with b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe not found: ID does not exist" containerID="b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe"
May 20 16:05:19.441930 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.441900    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe"} err="failed to get container status \"b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe\": rpc error: code = NotFound desc = could not find container \"b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe\": container with ID starting with b634cbfb6e73a40e5fbdad0e1b6fa16124dea9658e7b9d249e118e4490eb78fe not found: ID does not exist"
May 20 16:05:19.450128 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.450091    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"]
May 20 16:05:19.454185 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.454164    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a8175b-predictor-5fb455746d-2hm67"]
May 20 16:05:19.890826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:19.890795    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" path="/var/lib/kubelet/pods/bb987d2e-a0ae-400d-88d8-637a6b681b35/volumes"
May 20 16:05:20.066558 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.066527    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"]
May 20 16:05:20.425584 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.425544    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf" podUID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerName="storage-initializer" containerID="cri-o://47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712" gracePeriod=30
May 20 16:05:20.715679 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715610    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"]
May 20 16:05:20.715883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715870    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="storage-initializer"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715884    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="storage-initializer"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715893    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae92600-44fb-4ba8-aa45-55bec06cd852" containerName="storage-initializer"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715900    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae92600-44fb-4ba8-aa45-55bec06cd852" containerName="storage-initializer"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715911    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae92600-44fb-4ba8-aa45-55bec06cd852" containerName="storage-initializer"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715917    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae92600-44fb-4ba8-aa45-55bec06cd852" containerName="storage-initializer"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715925    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715930    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715935    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kube-rbac-proxy"
May 20 16:05:20.715938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715941    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kube-rbac-proxy"
May 20 16:05:20.716237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715985    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kube-rbac-proxy"
May 20 16:05:20.716237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.715995    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ae92600-44fb-4ba8-aa45-55bec06cd852" containerName="storage-initializer"
May 20 16:05:20.716237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.716002    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ae92600-44fb-4ba8-aa45-55bec06cd852" containerName="storage-initializer"
May 20 16:05:20.716237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.716009    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb987d2e-a0ae-400d-88d8-637a6b681b35" containerName="kserve-container"
May 20 16:05:20.718828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.718807    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.721291 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.721267    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
May 20 16:05:20.721391 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.721363    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
May 20 16:05:20.722415 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.722398    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fxjhf\""
May 20 16:05:20.727521 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.727499    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"]
May 20 16:05:20.838950 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.838925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/821e6fc3-497c-418e-9b53-53dab2d9a820-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.839034 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.838974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkht\" (UniqueName: \"kubernetes.io/projected/821e6fc3-497c-418e-9b53-53dab2d9a820-kube-api-access-pxkht\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.839085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.839039    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/821e6fc3-497c-418e-9b53-53dab2d9a820-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.839148 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.839095    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/821e6fc3-497c-418e-9b53-53dab2d9a820-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.940342 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.940309    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkht\" (UniqueName: \"kubernetes.io/projected/821e6fc3-497c-418e-9b53-53dab2d9a820-kube-api-access-pxkht\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.940449 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.940349    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/821e6fc3-497c-418e-9b53-53dab2d9a820-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.940449 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.940381    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/821e6fc3-497c-418e-9b53-53dab2d9a820-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.940449 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.940406    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/821e6fc3-497c-418e-9b53-53dab2d9a820-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.940846 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.940825    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/821e6fc3-497c-418e-9b53-53dab2d9a820-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.941145 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.941091    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/821e6fc3-497c-418e-9b53-53dab2d9a820-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.943017 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.942997    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/821e6fc3-497c-418e-9b53-53dab2d9a820-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:20.950296 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:20.950273    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkht\" (UniqueName: \"kubernetes.io/projected/821e6fc3-497c-418e-9b53-53dab2d9a820-kube-api-access-pxkht\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:21.030225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:21.030199    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:21.160828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:21.160805    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"]
May 20 16:05:21.163359 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:05:21.163335    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821e6fc3_497c_418e_9b53_53dab2d9a820.slice/crio-2c462b364f83b9df9f3e9dc81db3f7e724fc0e92992d306dde98b9c237fa1fb4 WatchSource:0}: Error finding container 2c462b364f83b9df9f3e9dc81db3f7e724fc0e92992d306dde98b9c237fa1fb4: Status 404 returned error can't find the container with id 2c462b364f83b9df9f3e9dc81db3f7e724fc0e92992d306dde98b9c237fa1fb4
May 20 16:05:21.429200 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:21.429124    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" event={"ID":"821e6fc3-497c-418e-9b53-53dab2d9a820","Type":"ContainerStarted","Data":"92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934"}
May 20 16:05:21.429200 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:21.429164    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" event={"ID":"821e6fc3-497c-418e-9b53-53dab2d9a820","Type":"ContainerStarted","Data":"2c462b364f83b9df9f3e9dc81db3f7e724fc0e92992d306dde98b9c237fa1fb4"}
May 20 16:05:23.758950 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.758929    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf_020aa1f2-0e06-4309-a1b4-39ad7beddee0/storage-initializer/1.log"
May 20 16:05:23.759309 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.759296    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf_020aa1f2-0e06-4309-a1b4-39ad7beddee0/storage-initializer/0.log"
May 20 16:05:23.759375 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.759364    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:23.863317 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.863258    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmhxc\" (UniqueName: \"kubernetes.io/projected/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kube-api-access-rmhxc\") pod \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") "
May 20 16:05:23.863317 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.863294    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kserve-provision-location\") pod \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") "
May 20 16:05:23.863317 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.863315    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-cabundle-cert\") pod \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") "
May 20 16:05:23.863478 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.863347    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\") pod \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") "
May 20 16:05:23.863478 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.863365    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020aa1f2-0e06-4309-a1b4-39ad7beddee0-proxy-tls\") pod \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\" (UID: \"020aa1f2-0e06-4309-a1b4-39ad7beddee0\") "
May 20 16:05:23.863658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.863608    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "020aa1f2-0e06-4309-a1b4-39ad7beddee0" (UID: "020aa1f2-0e06-4309-a1b4-39ad7beddee0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:23.863738 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.863709    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "020aa1f2-0e06-4309-a1b4-39ad7beddee0" (UID: "020aa1f2-0e06-4309-a1b4-39ad7beddee0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:05:23.863781 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.863744    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config") pod "020aa1f2-0e06-4309-a1b4-39ad7beddee0" (UID: "020aa1f2-0e06-4309-a1b4-39ad7beddee0"). InnerVolumeSpecName "isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:05:23.865394 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.865370    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020aa1f2-0e06-4309-a1b4-39ad7beddee0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "020aa1f2-0e06-4309-a1b4-39ad7beddee0" (UID: "020aa1f2-0e06-4309-a1b4-39ad7beddee0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:05:23.865545 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.865471    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kube-api-access-rmhxc" (OuterVolumeSpecName: "kube-api-access-rmhxc") pod "020aa1f2-0e06-4309-a1b4-39ad7beddee0" (UID: "020aa1f2-0e06-4309-a1b4-39ad7beddee0"). InnerVolumeSpecName "kube-api-access-rmhxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:05:23.964516 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.964492    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhxc\" (UniqueName: \"kubernetes.io/projected/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kube-api-access-rmhxc\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:23.964516 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.964512    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020aa1f2-0e06-4309-a1b4-39ad7beddee0-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:23.964625 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.964521    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-cabundle-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:23.964625 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.964531    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/020aa1f2-0e06-4309-a1b4-39ad7beddee0-isvc-init-fail-3cec1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:23.964625 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:23.964541    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020aa1f2-0e06-4309-a1b4-39ad7beddee0-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:05:24.438495 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.438471    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf_020aa1f2-0e06-4309-a1b4-39ad7beddee0/storage-initializer/1.log"
May 20 16:05:24.438900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.438881    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf_020aa1f2-0e06-4309-a1b4-39ad7beddee0/storage-initializer/0.log"
May 20 16:05:24.439019 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.438926    2556 generic.go:358] "Generic (PLEG): container finished" podID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerID="47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712" exitCode=1
May 20 16:05:24.439068 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.439012    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"
May 20 16:05:24.439068 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.439017    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf" event={"ID":"020aa1f2-0e06-4309-a1b4-39ad7beddee0","Type":"ContainerDied","Data":"47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712"}
May 20 16:05:24.439068 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.439060    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf" event={"ID":"020aa1f2-0e06-4309-a1b4-39ad7beddee0","Type":"ContainerDied","Data":"b6875f52d16350a215163cda57713bf44dd641e620062e6decb188f53e3f2587"}
May 20 16:05:24.439210 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.439076    2556 scope.go:117] "RemoveContainer" containerID="47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712"
May 20 16:05:24.446896 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.446876    2556 scope.go:117] "RemoveContainer" containerID="7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590"
May 20 16:05:24.453795 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.453775    2556 scope.go:117] "RemoveContainer" containerID="47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712"
May 20 16:05:24.454046 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:05:24.454026    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712\": container with ID starting with 47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712 not found: ID does not exist" containerID="47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712"
May 20 16:05:24.454118 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.454055    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712"} err="failed to get container status \"47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712\": rpc error: code = NotFound desc = could not find container \"47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712\": container with ID starting with 47aac298b38d2f7eb8982e97fe15c03022c59b9e38d5ac93e7e12d6b9d9a6712 not found: ID does not exist"
May 20 16:05:24.454118 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.454074    2556 scope.go:117] "RemoveContainer" containerID="7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590"
May 20 16:05:24.454349 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:05:24.454328    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590\": container with ID starting with 7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590 not found: ID does not exist" containerID="7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590"
May 20 16:05:24.454411 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.454359    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590"} err="failed to get container status \"7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590\": rpc error: code = NotFound desc = could not find container \"7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590\": container with ID starting with 7a048b4c8b313721a5ab2f8562cdea002369d34ddcaaeaa9c6544a958b3b3590 not found: ID does not exist"
May 20 16:05:24.473295 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.473269    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"]
May 20 16:05:24.476877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:24.476856    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3cec1b-predictor-6795dbd59f-9ptgf"]
May 20 16:05:25.443324 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:25.443248    2556 generic.go:358] "Generic (PLEG): container finished" podID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerID="92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934" exitCode=0
May 20 16:05:25.443755 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:25.443333    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" event={"ID":"821e6fc3-497c-418e-9b53-53dab2d9a820","Type":"ContainerDied","Data":"92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934"}
May 20 16:05:25.891039 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:25.891008    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" path="/var/lib/kubelet/pods/020aa1f2-0e06-4309-a1b4-39ad7beddee0/volumes"
May 20 16:05:50.532286 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:50.532246    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" event={"ID":"821e6fc3-497c-418e-9b53-53dab2d9a820","Type":"ContainerStarted","Data":"bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e"}
May 20 16:05:50.532286 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:50.532290    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" event={"ID":"821e6fc3-497c-418e-9b53-53dab2d9a820","Type":"ContainerStarted","Data":"94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2"}
May 20 16:05:50.532795 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:50.532491    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:50.532795 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:50.532521    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:50.533876 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:50.533854    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:05:50.551476 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:50.551431    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podStartSLOduration=6.312762017 podStartE2EDuration="30.551420963s" podCreationTimestamp="2026-05-20 16:05:20 +0000 UTC" firstStartedPulling="2026-05-20 16:05:25.444739647 +0000 UTC m=+1968.135995556" lastFinishedPulling="2026-05-20 16:05:49.683398591 +0000 UTC m=+1992.374654502" observedRunningTime="2026-05-20 16:05:50.550180299 +0000 UTC m=+1993.241436232" watchObservedRunningTime="2026-05-20 16:05:50.551420963 +0000 UTC m=+1993.242676890"
May 20 16:05:51.534981 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:51.534943    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:05:56.539246 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:56.539217    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:05:56.539788 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:05:56.539761    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:06:06.540226 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:06:06.540186    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:06:16.540604 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:06:16.540562    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:06:26.539662 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:06:26.539619    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:06:36.539805 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:06:36.539765    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:06:46.540353 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:06:46.540310    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:06:56.540082 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:06:56.539987    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 20 16:07:05.890736 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:05.890704    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:07:10.371066 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.371039    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"]
May 20 16:07:10.371454 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.371382    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container" containerID="cri-o://94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2" gracePeriod=30
May 20 16:07:10.371530 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.371453    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kube-rbac-proxy" containerID="cri-o://bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e" gracePeriod=30
May 20 16:07:10.510762 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.510734    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"]
May 20 16:07:10.511020 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.511000    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerName="storage-initializer"
May 20 16:07:10.511020 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.511017    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerName="storage-initializer"
May 20 16:07:10.511225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.511039    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerName="storage-initializer"
May 20 16:07:10.511225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.511047    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerName="storage-initializer"
May 20 16:07:10.511225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.511137    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerName="storage-initializer"
May 20 16:07:10.511225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.511149    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="020aa1f2-0e06-4309-a1b4-39ad7beddee0" containerName="storage-initializer"
May 20 16:07:10.513135 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.513119    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.515555 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.515538    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
May 20 16:07:10.515740 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.515726    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
May 20 16:07:10.523452 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.523432    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"]
May 20 16:07:10.553436 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.553411    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0e5891-a832-4948-a9b3-fa98366b0336-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.553529 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.553465    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.553580 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.553551    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbn4\" (UniqueName: \"kubernetes.io/projected/4d0e5891-a832-4948-a9b3-fa98366b0336-kube-api-access-tbbn4\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.553616 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.553596    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0e5891-a832-4948-a9b3-fa98366b0336-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.654371 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.654309    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.654371 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.654351    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbn4\" (UniqueName: \"kubernetes.io/projected/4d0e5891-a832-4948-a9b3-fa98366b0336-kube-api-access-tbbn4\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.654501 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.654371    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0e5891-a832-4948-a9b3-fa98366b0336-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.654501 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:07:10.654441    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
May 20 16:07:10.654501 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:07:10.654489    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls podName:4d0e5891-a832-4948-a9b3-fa98366b0336 nodeName:}" failed. No retries permitted until 2026-05-20 16:07:11.154475583 +0000 UTC m=+2073.845731488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls") pod "isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" (UID: "4d0e5891-a832-4948-a9b3-fa98366b0336") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
May 20 16:07:10.654660 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.654549    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0e5891-a832-4948-a9b3-fa98366b0336-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.654725 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.654667    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0e5891-a832-4948-a9b3-fa98366b0336-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.655087 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.655065    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0e5891-a832-4948-a9b3-fa98366b0336-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.663398 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.663378    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbn4\" (UniqueName: \"kubernetes.io/projected/4d0e5891-a832-4948-a9b3-fa98366b0336-kube-api-access-tbbn4\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:10.750118 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.750073    2556 generic.go:358] "Generic (PLEG): container finished" podID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerID="bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e" exitCode=2
May 20 16:07:10.750223 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:10.750151    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" event={"ID":"821e6fc3-497c-418e-9b53-53dab2d9a820","Type":"ContainerDied","Data":"bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e"}
May 20 16:07:11.158016 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:11.157986    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:11.160269 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:11.160240    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:11.423302 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:11.423199    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:11.535785 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:11.535748    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
May 20 16:07:11.544435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:11.544413    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"]
May 20 16:07:11.546911 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:07:11.546886    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0e5891_a832_4948_a9b3_fa98366b0336.slice/crio-15dd10ee2d623f7e9641a5fab83fd6ae560fb776077db52a5bf9c841ae542a2a WatchSource:0}: Error finding container 15dd10ee2d623f7e9641a5fab83fd6ae560fb776077db52a5bf9c841ae542a2a: Status 404 returned error can't find the container with id 15dd10ee2d623f7e9641a5fab83fd6ae560fb776077db52a5bf9c841ae542a2a
May 20 16:07:11.756277 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:11.756247    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" event={"ID":"4d0e5891-a832-4948-a9b3-fa98366b0336","Type":"ContainerStarted","Data":"57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b"}
May 20 16:07:11.756452 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:11.756287    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" event={"ID":"4d0e5891-a832-4948-a9b3-fa98366b0336","Type":"ContainerStarted","Data":"15dd10ee2d623f7e9641a5fab83fd6ae560fb776077db52a5bf9c841ae542a2a"}
May 20 16:07:14.711138 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.711117    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:07:14.766688 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.766622    2556 generic.go:358] "Generic (PLEG): container finished" podID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerID="94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2" exitCode=0
May 20 16:07:14.766789 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.766699    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" event={"ID":"821e6fc3-497c-418e-9b53-53dab2d9a820","Type":"ContainerDied","Data":"94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2"}
May 20 16:07:14.766789 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.766711    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"
May 20 16:07:14.766789 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.766734    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh" event={"ID":"821e6fc3-497c-418e-9b53-53dab2d9a820","Type":"ContainerDied","Data":"2c462b364f83b9df9f3e9dc81db3f7e724fc0e92992d306dde98b9c237fa1fb4"}
May 20 16:07:14.766789 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.766751    2556 scope.go:117] "RemoveContainer" containerID="bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e"
May 20 16:07:14.773737 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.773720    2556 scope.go:117] "RemoveContainer" containerID="94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2"
May 20 16:07:14.780023 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.780010    2556 scope.go:117] "RemoveContainer" containerID="92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934"
May 20 16:07:14.786287 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786186    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/821e6fc3-497c-418e-9b53-53dab2d9a820-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"821e6fc3-497c-418e-9b53-53dab2d9a820\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") "
May 20 16:07:14.786287 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786217    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxkht\" (UniqueName: \"kubernetes.io/projected/821e6fc3-497c-418e-9b53-53dab2d9a820-kube-api-access-pxkht\") pod \"821e6fc3-497c-418e-9b53-53dab2d9a820\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") "
May 20 16:07:14.786376 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786297    2556 scope.go:117] "RemoveContainer" containerID="bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e"
May 20 16:07:14.786376 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786302    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/821e6fc3-497c-418e-9b53-53dab2d9a820-kserve-provision-location\") pod \"821e6fc3-497c-418e-9b53-53dab2d9a820\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") "
May 20 16:07:14.786451 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786409    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/821e6fc3-497c-418e-9b53-53dab2d9a820-proxy-tls\") pod \"821e6fc3-497c-418e-9b53-53dab2d9a820\" (UID: \"821e6fc3-497c-418e-9b53-53dab2d9a820\") "
May 20 16:07:14.786576 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786549    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821e6fc3-497c-418e-9b53-53dab2d9a820-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "821e6fc3-497c-418e-9b53-53dab2d9a820" (UID: "821e6fc3-497c-418e-9b53-53dab2d9a820"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:07:14.786670 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786575    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821e6fc3-497c-418e-9b53-53dab2d9a820-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "821e6fc3-497c-418e-9b53-53dab2d9a820" (UID: "821e6fc3-497c-418e-9b53-53dab2d9a820"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:07:14.786670 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786651    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/821e6fc3-497c-418e-9b53-53dab2d9a820-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:07:14.786779 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786673    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/821e6fc3-497c-418e-9b53-53dab2d9a820-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:07:14.786865 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:07:14.786831    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e\": container with ID starting with bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e not found: ID does not exist" containerID="bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e"
May 20 16:07:14.786968 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786876    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e"} err="failed to get container status \"bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e\": rpc error: code = NotFound desc = could not find container \"bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e\": container with ID starting with bb2cbd27b6f30ee524efc4620bcfc9d37619dbba2cdc34123d2099847be05f8e not found: ID does not exist"
May 20 16:07:14.786968 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.786900    2556 scope.go:117] "RemoveContainer" containerID="94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2"
May 20 16:07:14.787186 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:07:14.787165    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2\": container with ID starting with 94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2 not found: ID does not exist" containerID="94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2"
May 20 16:07:14.787263 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.787198    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2"} err="failed to get container status \"94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2\": rpc error: code = NotFound desc = could not find container \"94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2\": container with ID starting with 94190b38247796a6f6172365620ace91da0d592f569dec3342d45c89dcc0f5a2 not found: ID does not exist"
May 20 16:07:14.787263 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.787222    2556 scope.go:117] "RemoveContainer" containerID="92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934"
May 20 16:07:14.787447 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:07:14.787422    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934\": container with ID starting with 92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934 not found: ID does not exist" containerID="92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934"
May 20 16:07:14.787495 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.787459    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934"} err="failed to get container status \"92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934\": rpc error: code = NotFound desc = could not find container \"92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934\": container with ID starting with 92118ab1bcda564fb22f982bf6c028e1f9b7651a6bfb3b5f82fdb3dc9f09b934 not found: ID does not exist"
May 20 16:07:14.788355 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.788333    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821e6fc3-497c-418e-9b53-53dab2d9a820-kube-api-access-pxkht" (OuterVolumeSpecName: "kube-api-access-pxkht") pod "821e6fc3-497c-418e-9b53-53dab2d9a820" (UID: "821e6fc3-497c-418e-9b53-53dab2d9a820"). InnerVolumeSpecName "kube-api-access-pxkht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:07:14.788522 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.788506    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821e6fc3-497c-418e-9b53-53dab2d9a820-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "821e6fc3-497c-418e-9b53-53dab2d9a820" (UID: "821e6fc3-497c-418e-9b53-53dab2d9a820"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:07:14.887047 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.887014    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/821e6fc3-497c-418e-9b53-53dab2d9a820-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:07:14.887047 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:14.887040    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkht\" (UniqueName: \"kubernetes.io/projected/821e6fc3-497c-418e-9b53-53dab2d9a820-kube-api-access-pxkht\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:07:15.089934 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:15.089908    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"]
May 20 16:07:15.097653 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:15.097628    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-4qsgh"]
May 20 16:07:15.771677 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:15.771596    2556 generic.go:358] "Generic (PLEG): container finished" podID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerID="57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b" exitCode=0
May 20 16:07:15.772000 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:15.771670    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" event={"ID":"4d0e5891-a832-4948-a9b3-fa98366b0336","Type":"ContainerDied","Data":"57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b"}
May 20 16:07:15.891095 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:15.891072    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" path="/var/lib/kubelet/pods/821e6fc3-497c-418e-9b53-53dab2d9a820/volumes"
May 20 16:07:16.776585 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:16.776558    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" event={"ID":"4d0e5891-a832-4948-a9b3-fa98366b0336","Type":"ContainerStarted","Data":"9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0"}
May 20 16:07:16.776585 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:16.776588    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" event={"ID":"4d0e5891-a832-4948-a9b3-fa98366b0336","Type":"ContainerStarted","Data":"8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f"}
May 20 16:07:16.777038 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:16.776918    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:16.777038 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:16.776941    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:16.778164 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:16.778138    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:07:16.796860 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:16.796824    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podStartSLOduration=6.796812764 podStartE2EDuration="6.796812764s" podCreationTimestamp="2026-05-20 16:07:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:07:16.794802497 +0000 UTC m=+2079.486058425" watchObservedRunningTime="2026-05-20 16:07:16.796812764 +0000 UTC m=+2079.488068692"
May 20 16:07:17.783213 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:17.783170    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:07:22.785420 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:22.785379    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:07:22.786019 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:22.785990    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:07:32.786177 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:32.786123    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:07:42.786751 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:42.786711    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:07:52.786756 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:07:52.786708    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:08:02.786035 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:02.785999    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:08:12.785903 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:12.785862    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:08:22.786480 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:22.786437    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
May 20 16:08:27.892324 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:27.892230    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:08:30.579358 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.579316    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"]
May 20 16:08:30.579852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.579613    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container" containerID="cri-o://8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f" gracePeriod=30
May 20 16:08:30.579852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.579684    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kube-rbac-proxy" containerID="cri-o://9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0" gracePeriod=30
May 20 16:08:30.649981 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.649953    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"]
May 20 16:08:30.650223 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.650210    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="storage-initializer"
May 20 16:08:30.650283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.650225    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="storage-initializer"
May 20 16:08:30.650283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.650237    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kube-rbac-proxy"
May 20 16:08:30.650283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.650243    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kube-rbac-proxy"
May 20 16:08:30.650283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.650251    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container"
May 20 16:08:30.650283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.650256    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container"
May 20 16:08:30.650459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.650303    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kserve-container"
May 20 16:08:30.650459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.650311    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="821e6fc3-497c-418e-9b53-53dab2d9a820" containerName="kube-rbac-proxy"
May 20 16:08:30.653334 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.653320    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.655745 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.655726    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
May 20 16:08:30.655925 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.655907    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
May 20 16:08:30.661479 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.661457    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"]
May 20 16:08:30.694645 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.694625    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e239801f-cd70-49b9-979f-9ba3172a4311-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.694761 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.694679    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e239801f-cd70-49b9-979f-9ba3172a4311-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.694761 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.694745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.694869 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.694772    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chvq\" (UniqueName: \"kubernetes.io/projected/e239801f-cd70-49b9-979f-9ba3172a4311-kube-api-access-9chvq\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.796020 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.795997    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e239801f-cd70-49b9-979f-9ba3172a4311-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.796141 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.796039    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e239801f-cd70-49b9-979f-9ba3172a4311-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.796141 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.796077    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.796141 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.796126    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9chvq\" (UniqueName: \"kubernetes.io/projected/e239801f-cd70-49b9-979f-9ba3172a4311-kube-api-access-9chvq\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.796303 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:08:30.796244    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
May 20 16:08:30.796303 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:08:30.796300    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls podName:e239801f-cd70-49b9-979f-9ba3172a4311 nodeName:}" failed. No retries permitted until 2026-05-20 16:08:31.296281994 +0000 UTC m=+2153.987537901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls") pod "isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" (UID: "e239801f-cd70-49b9-979f-9ba3172a4311") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
May 20 16:08:30.796513 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.796495    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e239801f-cd70-49b9-979f-9ba3172a4311-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.796790 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.796767    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e239801f-cd70-49b9-979f-9ba3172a4311-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.804888 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.804860    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chvq\" (UniqueName: \"kubernetes.io/projected/e239801f-cd70-49b9-979f-9ba3172a4311-kube-api-access-9chvq\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:30.978743 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.978637    2556 generic.go:358] "Generic (PLEG): container finished" podID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerID="9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0" exitCode=2
May 20 16:08:30.978743 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:30.978695    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" event={"ID":"4d0e5891-a832-4948-a9b3-fa98366b0336","Type":"ContainerDied","Data":"9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0"}
May 20 16:08:31.300083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:31.300059    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:31.302474 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:31.302451    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:31.563927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:31.563856    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:31.689342 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:31.689296    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"]
May 20 16:08:31.691567 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:08:31.691540    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode239801f_cd70_49b9_979f_9ba3172a4311.slice/crio-fffc8356f2c1f754dece260d3fc07c7a4cf8a9419314850cebc6e71fec3a665b WatchSource:0}: Error finding container fffc8356f2c1f754dece260d3fc07c7a4cf8a9419314850cebc6e71fec3a665b: Status 404 returned error can't find the container with id fffc8356f2c1f754dece260d3fc07c7a4cf8a9419314850cebc6e71fec3a665b
May 20 16:08:31.982785 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:31.982702    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" event={"ID":"e239801f-cd70-49b9-979f-9ba3172a4311","Type":"ContainerStarted","Data":"c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d"}
May 20 16:08:31.982785 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:31.982745    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" event={"ID":"e239801f-cd70-49b9-979f-9ba3172a4311","Type":"ContainerStarted","Data":"fffc8356f2c1f754dece260d3fc07c7a4cf8a9419314850cebc6e71fec3a665b"}
May 20 16:08:32.782668 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:32.782631    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
May 20 16:08:34.615761 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.615740    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:08:34.627901 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.627882    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0e5891-a832-4948-a9b3-fa98366b0336-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"4d0e5891-a832-4948-a9b3-fa98366b0336\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") "
May 20 16:08:34.627965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.627917    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0e5891-a832-4948-a9b3-fa98366b0336-kserve-provision-location\") pod \"4d0e5891-a832-4948-a9b3-fa98366b0336\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") "
May 20 16:08:34.628053 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.628034    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls\") pod \"4d0e5891-a832-4948-a9b3-fa98366b0336\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") "
May 20 16:08:34.628095 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.628074    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbbn4\" (UniqueName: \"kubernetes.io/projected/4d0e5891-a832-4948-a9b3-fa98366b0336-kube-api-access-tbbn4\") pod \"4d0e5891-a832-4948-a9b3-fa98366b0336\" (UID: \"4d0e5891-a832-4948-a9b3-fa98366b0336\") "
May 20 16:08:34.628240 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.628217    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0e5891-a832-4948-a9b3-fa98366b0336-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d0e5891-a832-4948-a9b3-fa98366b0336" (UID: "4d0e5891-a832-4948-a9b3-fa98366b0336"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:08:34.628315 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.628252    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0e5891-a832-4948-a9b3-fa98366b0336-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "4d0e5891-a832-4948-a9b3-fa98366b0336" (UID: "4d0e5891-a832-4948-a9b3-fa98366b0336"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:08:34.628383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.628326    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0e5891-a832-4948-a9b3-fa98366b0336-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:08:34.628383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.628344    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0e5891-a832-4948-a9b3-fa98366b0336-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:08:34.629863 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.629837    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d0e5891-a832-4948-a9b3-fa98366b0336" (UID: "4d0e5891-a832-4948-a9b3-fa98366b0336"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:08:34.629967 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.629948    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0e5891-a832-4948-a9b3-fa98366b0336-kube-api-access-tbbn4" (OuterVolumeSpecName: "kube-api-access-tbbn4") pod "4d0e5891-a832-4948-a9b3-fa98366b0336" (UID: "4d0e5891-a832-4948-a9b3-fa98366b0336"). InnerVolumeSpecName "kube-api-access-tbbn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:08:34.729542 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.729491    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0e5891-a832-4948-a9b3-fa98366b0336-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:08:34.729542 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.729513    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbn4\" (UniqueName: \"kubernetes.io/projected/4d0e5891-a832-4948-a9b3-fa98366b0336-kube-api-access-tbbn4\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:08:34.991977 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.991943    2556 generic.go:358] "Generic (PLEG): container finished" podID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerID="8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f" exitCode=0
May 20 16:08:34.992143 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.992031    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"
May 20 16:08:34.992143 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.992024    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" event={"ID":"4d0e5891-a832-4948-a9b3-fa98366b0336","Type":"ContainerDied","Data":"8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f"}
May 20 16:08:34.992269 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.992155    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5" event={"ID":"4d0e5891-a832-4948-a9b3-fa98366b0336","Type":"ContainerDied","Data":"15dd10ee2d623f7e9641a5fab83fd6ae560fb776077db52a5bf9c841ae542a2a"}
May 20 16:08:34.992269 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.992180    2556 scope.go:117] "RemoveContainer" containerID="9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0"
May 20 16:08:34.999957 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:34.999941    2556 scope.go:117] "RemoveContainer" containerID="8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f"
May 20 16:08:35.006617 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.006601    2556 scope.go:117] "RemoveContainer" containerID="57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b"
May 20 16:08:35.013209 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.013190    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"]
May 20 16:08:35.013386 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.013373    2556 scope.go:117] "RemoveContainer" containerID="9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0"
May 20 16:08:35.013632 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:08:35.013614    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0\": container with ID starting with 9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0 not found: ID does not exist" containerID="9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0"
May 20 16:08:35.013689 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.013639    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0"} err="failed to get container status \"9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0\": rpc error: code = NotFound desc = could not find container \"9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0\": container with ID starting with 9015ba0e01198017897ddc221b61290a725acfd612a39b8c0231e4de03ec0cb0 not found: ID does not exist"
May 20 16:08:35.013689 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.013657    2556 scope.go:117] "RemoveContainer" containerID="8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f"
May 20 16:08:35.013853 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:08:35.013838    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f\": container with ID starting with 8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f not found: ID does not exist" containerID="8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f"
May 20 16:08:35.013906 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.013855    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f"} err="failed to get container status \"8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f\": rpc error: code = NotFound desc = could not find container \"8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f\": container with ID starting with 8b6904d5972bd5d64a2d2d001adc3f967f4ad7985a9cce205efe957e170ad99f not found: ID does not exist"
May 20 16:08:35.013906 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.013867    2556 scope.go:117] "RemoveContainer" containerID="57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b"
May 20 16:08:35.014054 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:08:35.014036    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b\": container with ID starting with 57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b not found: ID does not exist" containerID="57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b"
May 20 16:08:35.014107 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.014061    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b"} err="failed to get container status \"57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b\": rpc error: code = NotFound desc = could not find container \"57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b\": container with ID starting with 57cfaad1924bb4752e0631cb6324b014572cf3f87c8cbfedb479a693a0ad066b not found: ID does not exist"
May 20 16:08:35.017584 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.017565    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-vwmz5"]
May 20 16:08:35.890666 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.890602    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" path="/var/lib/kubelet/pods/4d0e5891-a832-4948-a9b3-fa98366b0336/volumes"
May 20 16:08:35.997147 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.997117    2556 generic.go:358] "Generic (PLEG): container finished" podID="e239801f-cd70-49b9-979f-9ba3172a4311" containerID="c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d" exitCode=0
May 20 16:08:35.997289 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:35.997181    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" event={"ID":"e239801f-cd70-49b9-979f-9ba3172a4311","Type":"ContainerDied","Data":"c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d"}
May 20 16:08:37.002205 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:37.002169    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" event={"ID":"e239801f-cd70-49b9-979f-9ba3172a4311","Type":"ContainerStarted","Data":"323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b"}
May 20 16:08:37.002205 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:37.002209    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" event={"ID":"e239801f-cd70-49b9-979f-9ba3172a4311","Type":"ContainerStarted","Data":"fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac"}
May 20 16:08:37.002681 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:37.002423    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:37.020929 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:37.020884    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podStartSLOduration=7.02087045 podStartE2EDuration="7.02087045s" podCreationTimestamp="2026-05-20 16:08:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:08:37.019921221 +0000 UTC m=+2159.711177152" watchObservedRunningTime="2026-05-20 16:08:37.02087045 +0000 UTC m=+2159.712126378"
May 20 16:08:38.005025 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:38.004995    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:38.006442 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:38.006418    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:08:39.009410 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:39.009353    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:08:44.015812 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:44.015775    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:08:44.016620 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:44.016587    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:08:54.017315 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:08:54.017278    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:09:04.017227 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:09:04.017189    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:09:14.016750 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:09:14.016712    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:09:24.016839 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:09:24.016783    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:09:34.016592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:09:34.016552    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:09:44.017216 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:09:44.017172    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:09:54.017593 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:09:54.017558    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:10:00.865078 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.864986    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"]
May 20 16:10:00.865551 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.865328    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" containerID="cri-o://fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac" gracePeriod=30
May 20 16:10:00.865551 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.865384    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kube-rbac-proxy" containerID="cri-o://323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b" gracePeriod=30
May 20 16:10:00.981498 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981473    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"]
May 20 16:10:00.981746 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981733    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kube-rbac-proxy"
May 20 16:10:00.981799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981748    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kube-rbac-proxy"
May 20 16:10:00.981799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981759    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="storage-initializer"
May 20 16:10:00.981799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981764    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="storage-initializer"
May 20 16:10:00.981799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981780    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container"
May 20 16:10:00.981799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981785    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container"
May 20 16:10:00.981954 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981840    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kserve-container"
May 20 16:10:00.981954 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.981849    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="4d0e5891-a832-4948-a9b3-fa98366b0336" containerName="kube-rbac-proxy"
May 20 16:10:00.984571 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.984556    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:00.986708 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.986688    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
May 20 16:10:00.986941 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.986927    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
May 20 16:10:00.993225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:00.993204    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"]
May 20 16:10:01.134217 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.134128    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpsx\" (UniqueName: \"kubernetes.io/projected/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kube-api-access-8rpsx\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.134217 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.134201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.134411 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.134229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.134411 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.134305    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.234702 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.234675    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.234841 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.234723    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpsx\" (UniqueName: \"kubernetes.io/projected/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kube-api-access-8rpsx\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.234841 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.234726    2556 generic.go:358] "Generic (PLEG): container finished" podID="e239801f-cd70-49b9-979f-9ba3172a4311" containerID="323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b" exitCode=2
May 20 16:10:01.234841 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.234756    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.234841 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.234782    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.234841 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.234816    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" event={"ID":"e239801f-cd70-49b9-979f-9ba3172a4311","Type":"ContainerDied","Data":"323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b"}
May 20 16:10:01.235233 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.235211    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.235388 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.235370    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.237278 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.237259    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.243860 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.243842    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpsx\" (UniqueName: \"kubernetes.io/projected/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kube-api-access-8rpsx\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.295259 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.295241    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:01.414612 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.414548    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"]
May 20 16:10:01.417583 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:10:01.417558    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf973828_8ae5_4ead_99c1_f6ec83d5aa3f.slice/crio-c844d34984b7b9ddcdc94f80cb19bf8aaf4f0f2a734c177832d50403b6128a3a WatchSource:0}: Error finding container c844d34984b7b9ddcdc94f80cb19bf8aaf4f0f2a734c177832d50403b6128a3a: Status 404 returned error can't find the container with id c844d34984b7b9ddcdc94f80cb19bf8aaf4f0f2a734c177832d50403b6128a3a
May 20 16:10:01.419288 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:01.419272    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:10:02.238933 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:02.238902    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" event={"ID":"af973828-8ae5-4ead-99c1-f6ec83d5aa3f","Type":"ContainerStarted","Data":"72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c"}
May 20 16:10:02.238933 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:02.238935    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" event={"ID":"af973828-8ae5-4ead-99c1-f6ec83d5aa3f","Type":"ContainerStarted","Data":"c844d34984b7b9ddcdc94f80cb19bf8aaf4f0f2a734c177832d50403b6128a3a"}
May 20 16:10:04.010116 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:04.010067    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
May 20 16:10:04.016796 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:04.016753    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
May 20 16:10:06.249891 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:06.249859    2556 generic.go:358] "Generic (PLEG): container finished" podID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerID="72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c" exitCode=0
May 20 16:10:06.250299 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:06.249934    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" event={"ID":"af973828-8ae5-4ead-99c1-f6ec83d5aa3f","Type":"ContainerDied","Data":"72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c"}
May 20 16:10:07.253943 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:07.253910    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" event={"ID":"af973828-8ae5-4ead-99c1-f6ec83d5aa3f","Type":"ContainerStarted","Data":"2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236"}
May 20 16:10:07.253943 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:07.253948    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" event={"ID":"af973828-8ae5-4ead-99c1-f6ec83d5aa3f","Type":"ContainerStarted","Data":"10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2"}
May 20 16:10:07.275763 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:07.275719    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podStartSLOduration=7.275706167 podStartE2EDuration="7.275706167s" podCreationTimestamp="2026-05-20 16:10:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:10:07.274360201 +0000 UTC m=+2249.965616187" watchObservedRunningTime="2026-05-20 16:10:07.275706167 +0000 UTC m=+2249.966962092"
May 20 16:10:08.404391 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.404363    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:10:08.484293 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.484271    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e239801f-cd70-49b9-979f-9ba3172a4311-kserve-provision-location\") pod \"e239801f-cd70-49b9-979f-9ba3172a4311\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") "
May 20 16:10:08.484435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.484305    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e239801f-cd70-49b9-979f-9ba3172a4311-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"e239801f-cd70-49b9-979f-9ba3172a4311\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") "
May 20 16:10:08.484435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.484334    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls\") pod \"e239801f-cd70-49b9-979f-9ba3172a4311\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") "
May 20 16:10:08.484435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.484371    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9chvq\" (UniqueName: \"kubernetes.io/projected/e239801f-cd70-49b9-979f-9ba3172a4311-kube-api-access-9chvq\") pod \"e239801f-cd70-49b9-979f-9ba3172a4311\" (UID: \"e239801f-cd70-49b9-979f-9ba3172a4311\") "
May 20 16:10:08.484655 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.484633    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e239801f-cd70-49b9-979f-9ba3172a4311-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e239801f-cd70-49b9-979f-9ba3172a4311" (UID: "e239801f-cd70-49b9-979f-9ba3172a4311"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:08.484697 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.484677    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e239801f-cd70-49b9-979f-9ba3172a4311-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "e239801f-cd70-49b9-979f-9ba3172a4311" (UID: "e239801f-cd70-49b9-979f-9ba3172a4311"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:10:08.486455 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.486426    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e239801f-cd70-49b9-979f-9ba3172a4311" (UID: "e239801f-cd70-49b9-979f-9ba3172a4311"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:10:08.486576 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.486469    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e239801f-cd70-49b9-979f-9ba3172a4311-kube-api-access-9chvq" (OuterVolumeSpecName: "kube-api-access-9chvq") pod "e239801f-cd70-49b9-979f-9ba3172a4311" (UID: "e239801f-cd70-49b9-979f-9ba3172a4311"). InnerVolumeSpecName "kube-api-access-9chvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:10:08.584971 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.584931    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e239801f-cd70-49b9-979f-9ba3172a4311-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:10:08.584971 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.584963    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9chvq\" (UniqueName: \"kubernetes.io/projected/e239801f-cd70-49b9-979f-9ba3172a4311-kube-api-access-9chvq\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:10:08.584971 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.584973    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e239801f-cd70-49b9-979f-9ba3172a4311-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:10:08.585236 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:08.584983    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e239801f-cd70-49b9-979f-9ba3172a4311-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:10:09.262911 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.262877    2556 generic.go:358] "Generic (PLEG): container finished" podID="e239801f-cd70-49b9-979f-9ba3172a4311" containerID="fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac" exitCode=0
May 20 16:10:09.263085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.262937    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" event={"ID":"e239801f-cd70-49b9-979f-9ba3172a4311","Type":"ContainerDied","Data":"fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac"}
May 20 16:10:09.263085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.262945    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"
May 20 16:10:09.263085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.262963    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j" event={"ID":"e239801f-cd70-49b9-979f-9ba3172a4311","Type":"ContainerDied","Data":"fffc8356f2c1f754dece260d3fc07c7a4cf8a9419314850cebc6e71fec3a665b"}
May 20 16:10:09.263085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.262978    2556 scope.go:117] "RemoveContainer" containerID="323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b"
May 20 16:10:09.270907 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.270888    2556 scope.go:117] "RemoveContainer" containerID="fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac"
May 20 16:10:09.277465 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.277450    2556 scope.go:117] "RemoveContainer" containerID="c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d"
May 20 16:10:09.283785 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.283568    2556 scope.go:117] "RemoveContainer" containerID="323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b"
May 20 16:10:09.283885 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:10:09.283865    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b\": container with ID starting with 323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b not found: ID does not exist" containerID="323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b"
May 20 16:10:09.283934 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.283895    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b"} err="failed to get container status \"323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b\": rpc error: code = NotFound desc = could not find container \"323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b\": container with ID starting with 323674306d94ccc7904e8b7177948241e34683d60645c9e6bc2d272ad731a00b not found: ID does not exist"
May 20 16:10:09.283934 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.283915    2556 scope.go:117] "RemoveContainer" containerID="fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac"
May 20 16:10:09.284219 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:10:09.284187    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac\": container with ID starting with fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac not found: ID does not exist" containerID="fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac"
May 20 16:10:09.284324 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.284223    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac"} err="failed to get container status \"fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac\": rpc error: code = NotFound desc = could not find container \"fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac\": container with ID starting with fcf9317405ad269daf4cdf0ffae7113e0ede39fb8f02dde88bf89a86a2e433ac not found: ID does not exist"
May 20 16:10:09.284324 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.284245    2556 scope.go:117] "RemoveContainer" containerID="c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d"
May 20 16:10:09.284539 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:10:09.284518    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d\": container with ID starting with c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d not found: ID does not exist" containerID="c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d"
May 20 16:10:09.284604 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.284549    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d"} err="failed to get container status \"c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d\": rpc error: code = NotFound desc = could not find container \"c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d\": container with ID starting with c3c9d1818d9968920831dd07a7aa110ee38ae5308738196952073e8d4622b83d not found: ID does not exist"
May 20 16:10:09.285484 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.285464    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"]
May 20 16:10:09.289951 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.289932    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-x6s5j"]
May 20 16:10:09.891581 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:09.891547    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" path="/var/lib/kubelet/pods/e239801f-cd70-49b9-979f-9ba3172a4311/volumes"
May 20 16:10:12.255262 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:12.255225    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:12.255694 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:12.255677    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:12.262718 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:12.262691    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:10:43.274947 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:43.274900    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.40:8080: connect: connection refused"
May 20 16:10:53.275630 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:10:53.275584    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.40:8080: connect: connection refused"
May 20 16:11:03.275181 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:03.275142    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.40:8080: connect: connection refused"
May 20 16:11:13.274785 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:13.274743    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.40:8080: connect: connection refused"
May 20 16:11:23.278963 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:23.278921    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:11:31.082709 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.082621    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"]
May 20 16:11:31.083264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.083046    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container" containerID="cri-o://10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2" gracePeriod=30
May 20 16:11:31.083264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.083144    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kube-rbac-proxy" containerID="cri-o://2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236" gracePeriod=30
May 20 16:11:31.180521 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180489    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"]
May 20 16:11:31.180790 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180773    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kube-rbac-proxy"
May 20 16:11:31.180868 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180792    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kube-rbac-proxy"
May 20 16:11:31.180868 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180819    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="storage-initializer"
May 20 16:11:31.180868 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180828    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="storage-initializer"
May 20 16:11:31.180868 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180847    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container"
May 20 16:11:31.180868 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180856    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container"
May 20 16:11:31.181156 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180923    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kube-rbac-proxy"
May 20 16:11:31.181156 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.180938    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="e239801f-cd70-49b9-979f-9ba3172a4311" containerName="kserve-container"
May 20 16:11:31.184001 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.183983    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.186304 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.186284    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
May 20 16:11:31.186402 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.186287    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
May 20 16:11:31.192186 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.192166    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"]
May 20 16:11:31.267637 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.267612    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1c3031-0492-4390-90ab-d1e9ad4c331c-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.267760 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.267658    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhn4l\" (UniqueName: \"kubernetes.io/projected/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kube-api-access-zhn4l\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.267760 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.267676    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.267760 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.267702    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1c3031-0492-4390-90ab-d1e9ad4c331c-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.368988 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.368908    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1c3031-0492-4390-90ab-d1e9ad4c331c-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.368988 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.368963    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn4l\" (UniqueName: \"kubernetes.io/projected/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kube-api-access-zhn4l\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.368988 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.368988    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.369298 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.369008    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1c3031-0492-4390-90ab-d1e9ad4c331c-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.369489 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.369465    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.369619 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.369601    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1c3031-0492-4390-90ab-d1e9ad4c331c-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.371396 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.371377    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1c3031-0492-4390-90ab-d1e9ad4c331c-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.377480 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.377457    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhn4l\" (UniqueName: \"kubernetes.io/projected/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kube-api-access-zhn4l\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.487537 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.487508    2556 generic.go:358] "Generic (PLEG): container finished" podID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerID="2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236" exitCode=2
May 20 16:11:31.487649 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.487565    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" event={"ID":"af973828-8ae5-4ead-99c1-f6ec83d5aa3f","Type":"ContainerDied","Data":"2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236"}
May 20 16:11:31.495807 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.495791    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:31.621244 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:31.621053    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"]
May 20 16:11:31.623697 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:11:31.623670    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1c3031_0492_4390_90ab_d1e9ad4c331c.slice/crio-901709595c5d3f4d7820a0d3ac91c3248a56d3338d355779a6c1a0d869272d63 WatchSource:0}: Error finding container 901709595c5d3f4d7820a0d3ac91c3248a56d3338d355779a6c1a0d869272d63: Status 404 returned error can't find the container with id 901709595c5d3f4d7820a0d3ac91c3248a56d3338d355779a6c1a0d869272d63
May 20 16:11:32.263374 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:32.263335    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
May 20 16:11:32.491433 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:32.491398    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" event={"ID":"2f1c3031-0492-4390-90ab-d1e9ad4c331c","Type":"ContainerStarted","Data":"cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc"}
May 20 16:11:32.491433 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:32.491435    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" event={"ID":"2f1c3031-0492-4390-90ab-d1e9ad4c331c","Type":"ContainerStarted","Data":"901709595c5d3f4d7820a0d3ac91c3248a56d3338d355779a6c1a0d869272d63"}
May 20 16:11:33.275276 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:33.275234    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.40:8080: connect: connection refused"
May 20 16:11:35.277746 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.277686    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:11:35.402376 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.402352    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") "
May 20 16:11:35.402509 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.402403    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rpsx\" (UniqueName: \"kubernetes.io/projected/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kube-api-access-8rpsx\") pod \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") "
May 20 16:11:35.402509 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.402426    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kserve-provision-location\") pod \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") "
May 20 16:11:35.402509 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.402482    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-proxy-tls\") pod \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\" (UID: \"af973828-8ae5-4ead-99c1-f6ec83d5aa3f\") "
May 20 16:11:35.402717 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.402694    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af973828-8ae5-4ead-99c1-f6ec83d5aa3f" (UID: "af973828-8ae5-4ead-99c1-f6ec83d5aa3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:11:35.402775 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.402729    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "af973828-8ae5-4ead-99c1-f6ec83d5aa3f" (UID: "af973828-8ae5-4ead-99c1-f6ec83d5aa3f"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:11:35.404505 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.404479    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af973828-8ae5-4ead-99c1-f6ec83d5aa3f" (UID: "af973828-8ae5-4ead-99c1-f6ec83d5aa3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:11:35.404605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.404582    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kube-api-access-8rpsx" (OuterVolumeSpecName: "kube-api-access-8rpsx") pod "af973828-8ae5-4ead-99c1-f6ec83d5aa3f" (UID: "af973828-8ae5-4ead-99c1-f6ec83d5aa3f"). InnerVolumeSpecName "kube-api-access-8rpsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:11:35.501022 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.500997    2556 generic.go:358] "Generic (PLEG): container finished" podID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerID="10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2" exitCode=0
May 20 16:11:35.501199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.501082    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" event={"ID":"af973828-8ae5-4ead-99c1-f6ec83d5aa3f","Type":"ContainerDied","Data":"10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2"}
May 20 16:11:35.501199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.501144    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq" event={"ID":"af973828-8ae5-4ead-99c1-f6ec83d5aa3f","Type":"ContainerDied","Data":"c844d34984b7b9ddcdc94f80cb19bf8aaf4f0f2a734c177832d50403b6128a3a"}
May 20 16:11:35.501199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.501093    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"
May 20 16:11:35.501335 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.501169    2556 scope.go:117] "RemoveContainer" containerID="2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236"
May 20 16:11:35.502605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.502584    2556 generic.go:358] "Generic (PLEG): container finished" podID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerID="cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc" exitCode=0
May 20 16:11:35.502714 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.502652    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" event={"ID":"2f1c3031-0492-4390-90ab-d1e9ad4c331c","Type":"ContainerDied","Data":"cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc"}
May 20 16:11:35.502970 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.502951    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:11:35.503021 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.502993    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpsx\" (UniqueName: \"kubernetes.io/projected/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kube-api-access-8rpsx\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:11:35.503068 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.503031    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:11:35.503068 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.503056    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af973828-8ae5-4ead-99c1-f6ec83d5aa3f-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:11:35.509259 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.509237    2556 scope.go:117] "RemoveContainer" containerID="10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2"
May 20 16:11:35.518087 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.518067    2556 scope.go:117] "RemoveContainer" containerID="72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c"
May 20 16:11:35.525541 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.525521    2556 scope.go:117] "RemoveContainer" containerID="2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236"
May 20 16:11:35.525855 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:11:35.525837    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236\": container with ID starting with 2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236 not found: ID does not exist" containerID="2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236"
May 20 16:11:35.525924 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.525862    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236"} err="failed to get container status \"2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236\": rpc error: code = NotFound desc = could not find container \"2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236\": container with ID starting with 2816ccd9bd228a89939fad95b90864ced170ed1a35c0601df1e17215bd864236 not found: ID does not exist"
May 20 16:11:35.525924 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.525879    2556 scope.go:117] "RemoveContainer" containerID="10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2"
May 20 16:11:35.526156 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:11:35.526137    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2\": container with ID starting with 10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2 not found: ID does not exist" containerID="10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2"
May 20 16:11:35.526209 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.526161    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2"} err="failed to get container status \"10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2\": rpc error: code = NotFound desc = could not find container \"10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2\": container with ID starting with 10aecb47fca726ab33297e81ec61d70c996ba0b814878c48bcc2da9fa09b84c2 not found: ID does not exist"
May 20 16:11:35.526209 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.526180    2556 scope.go:117] "RemoveContainer" containerID="72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c"
May 20 16:11:35.526425 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:11:35.526410    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c\": container with ID starting with 72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c not found: ID does not exist" containerID="72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c"
May 20 16:11:35.526477 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.526428    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c"} err="failed to get container status \"72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c\": rpc error: code = NotFound desc = could not find container \"72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c\": container with ID starting with 72f6455627d6c8dec571f8272595f43532e941afd55a03cfb77f42135a89821c not found: ID does not exist"
May 20 16:11:35.535512 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.535467    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"]
May 20 16:11:35.539496 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.539424    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-2tsgq"]
May 20 16:11:35.891034 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:35.890956    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" path="/var/lib/kubelet/pods/af973828-8ae5-4ead-99c1-f6ec83d5aa3f/volumes"
May 20 16:11:36.507489 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:36.507456    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" event={"ID":"2f1c3031-0492-4390-90ab-d1e9ad4c331c","Type":"ContainerStarted","Data":"cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6"}
May 20 16:11:36.507489 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:36.507492    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" event={"ID":"2f1c3031-0492-4390-90ab-d1e9ad4c331c","Type":"ContainerStarted","Data":"3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623"}
May 20 16:11:36.508034 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:36.507727    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:36.508034 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:36.507783    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:11:36.527511 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:36.527475    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podStartSLOduration=5.527463879 podStartE2EDuration="5.527463879s" podCreationTimestamp="2026-05-20 16:11:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:11:36.526567809 +0000 UTC m=+2339.217823750" watchObservedRunningTime="2026-05-20 16:11:36.527463879 +0000 UTC m=+2339.218719807"
May 20 16:11:42.516131 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:11:42.516085    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:12:12.516630 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:12:12.516586    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
May 20 16:12:22.516886 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:12:22.516849    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
May 20 16:12:32.517224 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:12:32.517177    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
May 20 16:12:42.517118 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:12:42.517063    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
May 20 16:12:42.887394 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:12:42.887303    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
May 20 16:12:52.891782 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:12:52.891747    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:13:01.311668 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.311576    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"]
May 20 16:13:01.312136 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.311951    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container" containerID="cri-o://3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623" gracePeriod=30
May 20 16:13:01.312136 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.312057    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kube-rbac-proxy" containerID="cri-o://cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6" gracePeriod=30
May 20 16:13:01.410150 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410122    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"]
May 20 16:13:01.410389 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410375    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kube-rbac-proxy"
May 20 16:13:01.410442 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410390    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kube-rbac-proxy"
May 20 16:13:01.410442 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410408    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container"
May 20 16:13:01.410442 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410416    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container"
May 20 16:13:01.410442 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410433    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="storage-initializer"
May 20 16:13:01.410442 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410439    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="storage-initializer"
May 20 16:13:01.410604 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410478    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kserve-container"
May 20 16:13:01.410604 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.410489    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="af973828-8ae5-4ead-99c1-f6ec83d5aa3f" containerName="kube-rbac-proxy"
May 20 16:13:01.413605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.413589    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.416383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.416353    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
May 20 16:13:01.416383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.416372    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
May 20 16:13:01.421138 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.421116    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"]
May 20 16:13:01.492892 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.492866    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4g8h\" (UniqueName: \"kubernetes.io/projected/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kube-api-access-d4g8h\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.492995 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.492905    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.492995 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.492926    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce58a5a-e1f7-47fd-946e-2a401a986fb7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.493088 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.492989    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.593881 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.593809    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d4g8h\" (UniqueName: \"kubernetes.io/projected/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kube-api-access-d4g8h\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.593881 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.593863    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.594050 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.593897    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce58a5a-e1f7-47fd-946e-2a401a986fb7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.594050 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.593929    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.594050 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:13:01.594018    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
May 20 16:13:01.594246 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:13:01.594089    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls podName:cce58a5a-e1f7-47fd-946e-2a401a986fb7 nodeName:}" failed. No retries permitted until 2026-05-20 16:13:02.094073352 +0000 UTC m=+2424.785329474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" (UID: "cce58a5a-e1f7-47fd-946e-2a401a986fb7") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
May 20 16:13:01.594387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.594369    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.594546 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.594527    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce58a5a-e1f7-47fd-946e-2a401a986fb7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.602461 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.602438    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4g8h\" (UniqueName: \"kubernetes.io/projected/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kube-api-access-d4g8h\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:01.750493 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.750468    2556 generic.go:358] "Generic (PLEG): container finished" podID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerID="cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6" exitCode=2
May 20 16:13:01.750588 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:01.750544    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" event={"ID":"2f1c3031-0492-4390-90ab-d1e9ad4c331c","Type":"ContainerDied","Data":"cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6"}
May 20 16:13:02.097018 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:02.096970    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:02.099436 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:02.099416    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:02.325085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:02.325051    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:02.448023 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:02.447989    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"]
May 20 16:13:02.451488 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:13:02.451455    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce58a5a_e1f7_47fd_946e_2a401a986fb7.slice/crio-9fd44a9a6985e4c66eca88de61e6ffc85b56bf65da5de303840ce66f34c12538 WatchSource:0}: Error finding container 9fd44a9a6985e4c66eca88de61e6ffc85b56bf65da5de303840ce66f34c12538: Status 404 returned error can't find the container with id 9fd44a9a6985e4c66eca88de61e6ffc85b56bf65da5de303840ce66f34c12538
May 20 16:13:02.511958 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:02.511925    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
May 20 16:13:02.755113 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:02.755077    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" event={"ID":"cce58a5a-e1f7-47fd-946e-2a401a986fb7","Type":"ContainerStarted","Data":"a2c4f84cc6b8e628db50e97165bf4fc85eb50f1bbc36fd68726400afa6707170"}
May 20 16:13:02.755286 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:02.755128    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" event={"ID":"cce58a5a-e1f7-47fd-946e-2a401a986fb7","Type":"ContainerStarted","Data":"9fd44a9a6985e4c66eca88de61e6ffc85b56bf65da5de303840ce66f34c12538"}
May 20 16:13:02.887384 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:02.887345    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
May 20 16:13:05.657819 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.657795    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:13:05.726239 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.726199    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kserve-provision-location\") pod \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") "
May 20 16:13:05.726239 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.726245    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhn4l\" (UniqueName: \"kubernetes.io/projected/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kube-api-access-zhn4l\") pod \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") "
May 20 16:13:05.726474 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.726287    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1c3031-0492-4390-90ab-d1e9ad4c331c-proxy-tls\") pod \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") "
May 20 16:13:05.726474 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.726333    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1c3031-0492-4390-90ab-d1e9ad4c331c-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\" (UID: \"2f1c3031-0492-4390-90ab-d1e9ad4c331c\") "
May 20 16:13:05.726585 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.726565    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f1c3031-0492-4390-90ab-d1e9ad4c331c" (UID: "2f1c3031-0492-4390-90ab-d1e9ad4c331c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:13:05.726800 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.726776    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1c3031-0492-4390-90ab-d1e9ad4c331c-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "2f1c3031-0492-4390-90ab-d1e9ad4c331c" (UID: "2f1c3031-0492-4390-90ab-d1e9ad4c331c"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:13:05.728427 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.728400    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1c3031-0492-4390-90ab-d1e9ad4c331c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f1c3031-0492-4390-90ab-d1e9ad4c331c" (UID: "2f1c3031-0492-4390-90ab-d1e9ad4c331c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:13:05.728427 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.728412    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kube-api-access-zhn4l" (OuterVolumeSpecName: "kube-api-access-zhn4l") pod "2f1c3031-0492-4390-90ab-d1e9ad4c331c" (UID: "2f1c3031-0492-4390-90ab-d1e9ad4c331c"). InnerVolumeSpecName "kube-api-access-zhn4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:13:05.764899 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.764812    2556 generic.go:358] "Generic (PLEG): container finished" podID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerID="3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623" exitCode=0
May 20 16:13:05.765046 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.764899    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" event={"ID":"2f1c3031-0492-4390-90ab-d1e9ad4c331c","Type":"ContainerDied","Data":"3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623"}
May 20 16:13:05.765046 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.764932    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4" event={"ID":"2f1c3031-0492-4390-90ab-d1e9ad4c331c","Type":"ContainerDied","Data":"901709595c5d3f4d7820a0d3ac91c3248a56d3338d355779a6c1a0d869272d63"}
May 20 16:13:05.765046 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.764947    2556 scope.go:117] "RemoveContainer" containerID="cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6"
May 20 16:13:05.765046 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.764905    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"
May 20 16:13:05.774090 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.774061    2556 scope.go:117] "RemoveContainer" containerID="3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623"
May 20 16:13:05.781320 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.781306    2556 scope.go:117] "RemoveContainer" containerID="cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc"
May 20 16:13:05.787988 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.787965    2556 scope.go:117] "RemoveContainer" containerID="cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6"
May 20 16:13:05.788360 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:13:05.788335    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6\": container with ID starting with cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6 not found: ID does not exist" containerID="cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6"
May 20 16:13:05.788468 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.788372    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6"} err="failed to get container status \"cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6\": rpc error: code = NotFound desc = could not find container \"cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6\": container with ID starting with cb3e80c134857b7025d2cfce0f47dfe9e5939fe785e3fcf5cf22ed680e9220a6 not found: ID does not exist"
May 20 16:13:05.788468 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.788396    2556 scope.go:117] "RemoveContainer" containerID="3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623"
May 20 16:13:05.788642 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:13:05.788624    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623\": container with ID starting with 3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623 not found: ID does not exist" containerID="3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623"
May 20 16:13:05.788693 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.788650    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623"} err="failed to get container status \"3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623\": rpc error: code = NotFound desc = could not find container \"3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623\": container with ID starting with 3713a67464c8dc6d613a3bb2464a47002c937cef1b8dd8e48b7c326388ba3623 not found: ID does not exist"
May 20 16:13:05.788693 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.788669    2556 scope.go:117] "RemoveContainer" containerID="cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc"
May 20 16:13:05.788924 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:13:05.788906    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc\": container with ID starting with cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc not found: ID does not exist" containerID="cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc"
May 20 16:13:05.788969 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.788927    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc"} err="failed to get container status \"cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc\": rpc error: code = NotFound desc = could not find container \"cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc\": container with ID starting with cb99b56d7288e758ad0e866c9d9fc44887e0c10746480162b2c05a2d704305cc not found: ID does not exist"
May 20 16:13:05.789589 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.789569    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"]
May 20 16:13:05.793047 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.793021    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-d97r4"]
May 20 16:13:05.827073 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.827038    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1c3031-0492-4390-90ab-d1e9ad4c331c-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:13:05.827073 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.827069    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1c3031-0492-4390-90ab-d1e9ad4c331c-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:13:05.827073 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.827080    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:13:05.827326 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.827091    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhn4l\" (UniqueName: \"kubernetes.io/projected/2f1c3031-0492-4390-90ab-d1e9ad4c331c-kube-api-access-zhn4l\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:13:05.890770 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:05.890731    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" path="/var/lib/kubelet/pods/2f1c3031-0492-4390-90ab-d1e9ad4c331c/volumes"
May 20 16:13:06.769492 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:06.769460    2556 generic.go:358] "Generic (PLEG): container finished" podID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerID="a2c4f84cc6b8e628db50e97165bf4fc85eb50f1bbc36fd68726400afa6707170" exitCode=0
May 20 16:13:06.769860 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:06.769495    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" event={"ID":"cce58a5a-e1f7-47fd-946e-2a401a986fb7","Type":"ContainerDied","Data":"a2c4f84cc6b8e628db50e97165bf4fc85eb50f1bbc36fd68726400afa6707170"}
May 20 16:13:07.774507 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:07.774470    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" event={"ID":"cce58a5a-e1f7-47fd-946e-2a401a986fb7","Type":"ContainerStarted","Data":"7a7e46206c170859638020e395dec390bd5b3fe3c78fe258bb666213b7af01b7"}
May 20 16:13:07.774999 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:07.774514    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" event={"ID":"cce58a5a-e1f7-47fd-946e-2a401a986fb7","Type":"ContainerStarted","Data":"69d7b6fe34497392c9df4d0b024d35bd389073c9fbc5cfca9bb00a5554291b4a"}
May 20 16:13:07.774999 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:07.774746    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:07.803743 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:07.803676    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podStartSLOduration=6.803658313 podStartE2EDuration="6.803658313s" podCreationTimestamp="2026-05-20 16:13:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:13:07.801908372 +0000 UTC m=+2430.493164300" watchObservedRunningTime="2026-05-20 16:13:07.803658313 +0000 UTC m=+2430.494914242"
May 20 16:13:08.777741 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:08.777716    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:14.786211 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:14.786182    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:13:44.787619 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:44.787573    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
May 20 16:13:54.787588 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:13:54.787547    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
May 20 16:14:04.787499 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:04.787461    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
May 20 16:14:14.787412 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:14.787367    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
May 20 16:14:24.790271 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:24.790219    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:14:31.521093 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:31.521006    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"]
May 20 16:14:31.521481 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:31.521424    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container" containerID="cri-o://69d7b6fe34497392c9df4d0b024d35bd389073c9fbc5cfca9bb00a5554291b4a" gracePeriod=30
May 20 16:14:31.521558 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:31.521463    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kube-rbac-proxy" containerID="cri-o://7a7e46206c170859638020e395dec390bd5b3fe3c78fe258bb666213b7af01b7" gracePeriod=30
May 20 16:14:32.004383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:32.004353    2556 generic.go:358] "Generic (PLEG): container finished" podID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerID="7a7e46206c170859638020e395dec390bd5b3fe3c78fe258bb666213b7af01b7" exitCode=2
May 20 16:14:32.004536 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:32.004419    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" event={"ID":"cce58a5a-e1f7-47fd-946e-2a401a986fb7","Type":"ContainerDied","Data":"7a7e46206c170859638020e395dec390bd5b3fe3c78fe258bb666213b7af01b7"}
May 20 16:14:33.742050 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742012    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"]
May 20 16:14:33.742437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742298    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container"
May 20 16:14:33.742437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742310    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container"
May 20 16:14:33.742437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742322    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kube-rbac-proxy"
May 20 16:14:33.742437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742328    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kube-rbac-proxy"
May 20 16:14:33.742437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742336    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="storage-initializer"
May 20 16:14:33.742437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742342    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="storage-initializer"
May 20 16:14:33.742437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742384    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kserve-container"
May 20 16:14:33.742437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.742393    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f1c3031-0492-4390-90ab-d1e9ad4c331c" containerName="kube-rbac-proxy"
May 20 16:14:33.745289 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.745270    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.748185 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.748165    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
May 20 16:14:33.748291 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.748168    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
May 20 16:14:33.760134 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.760087    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"]
May 20 16:14:33.825927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.825904    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb7ab-09a6-4bb6-986f-99bee04569f8-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.826021 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.825935    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kserve-provision-location\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.826021 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.825964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb7ab-09a6-4bb6-986f-99bee04569f8-proxy-tls\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.826115 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.826051    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5m7\" (UniqueName: \"kubernetes.io/projected/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kube-api-access-kx5m7\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.926899 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.926874    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb7ab-09a6-4bb6-986f-99bee04569f8-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.926997 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.926904    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kserve-provision-location\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.926997 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.926927    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb7ab-09a6-4bb6-986f-99bee04569f8-proxy-tls\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.926997 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.926959    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5m7\" (UniqueName: \"kubernetes.io/projected/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kube-api-access-kx5m7\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.927299 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.927283    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kserve-provision-location\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.927649 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.927632    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb7ab-09a6-4bb6-986f-99bee04569f8-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.929346 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.929330    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb7ab-09a6-4bb6-986f-99bee04569f8-proxy-tls\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:33.936047 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:33.936022    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5m7\" (UniqueName: \"kubernetes.io/projected/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kube-api-access-kx5m7\") pod \"isvc-sklearn-predictor-6b7bccc865-rfs5l\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:34.054529 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:34.054471    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:34.179012 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:34.178990    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"]
May 20 16:14:34.181492 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:14:34.181465    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccddb7ab_09a6_4bb6_986f_99bee04569f8.slice/crio-666f2fc7826e8a55c2aede0c1ecf5069fc704c334d1bb28404c1eec1dd59534b WatchSource:0}: Error finding container 666f2fc7826e8a55c2aede0c1ecf5069fc704c334d1bb28404c1eec1dd59534b: Status 404 returned error can't find the container with id 666f2fc7826e8a55c2aede0c1ecf5069fc704c334d1bb28404c1eec1dd59534b
May 20 16:14:34.781369 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:34.781327    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
May 20 16:14:34.786784 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:34.786749    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
May 20 16:14:35.014172 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:35.014136    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" event={"ID":"ccddb7ab-09a6-4bb6-986f-99bee04569f8","Type":"ContainerStarted","Data":"3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff"}
May 20 16:14:35.014172 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:35.014177    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" event={"ID":"ccddb7ab-09a6-4bb6-986f-99bee04569f8","Type":"ContainerStarted","Data":"666f2fc7826e8a55c2aede0c1ecf5069fc704c334d1bb28404c1eec1dd59534b"}
May 20 16:14:36.019044 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.019012    2556 generic.go:358] "Generic (PLEG): container finished" podID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerID="69d7b6fe34497392c9df4d0b024d35bd389073c9fbc5cfca9bb00a5554291b4a" exitCode=0
May 20 16:14:36.019431 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.019079    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" event={"ID":"cce58a5a-e1f7-47fd-946e-2a401a986fb7","Type":"ContainerDied","Data":"69d7b6fe34497392c9df4d0b024d35bd389073c9fbc5cfca9bb00a5554291b4a"}
May 20 16:14:36.156828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.156808    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:14:36.243578 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.243554    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls\") pod \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") "
May 20 16:14:36.243673 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.243597    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4g8h\" (UniqueName: \"kubernetes.io/projected/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kube-api-access-d4g8h\") pod \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") "
May 20 16:14:36.243673 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.243619    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce58a5a-e1f7-47fd-946e-2a401a986fb7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") "
May 20 16:14:36.243673 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.243663    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kserve-provision-location\") pod \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\" (UID: \"cce58a5a-e1f7-47fd-946e-2a401a986fb7\") "
May 20 16:14:36.243979 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.243951    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce58a5a-e1f7-47fd-946e-2a401a986fb7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "cce58a5a-e1f7-47fd-946e-2a401a986fb7" (UID: "cce58a5a-e1f7-47fd-946e-2a401a986fb7"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:14:36.244087 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.244029    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cce58a5a-e1f7-47fd-946e-2a401a986fb7" (UID: "cce58a5a-e1f7-47fd-946e-2a401a986fb7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:36.245619 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.245597    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cce58a5a-e1f7-47fd-946e-2a401a986fb7" (UID: "cce58a5a-e1f7-47fd-946e-2a401a986fb7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:14:36.245619 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.245608    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kube-api-access-d4g8h" (OuterVolumeSpecName: "kube-api-access-d4g8h") pod "cce58a5a-e1f7-47fd-946e-2a401a986fb7" (UID: "cce58a5a-e1f7-47fd-946e-2a401a986fb7"). InnerVolumeSpecName "kube-api-access-d4g8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:14:36.344726 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.344659    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce58a5a-e1f7-47fd-946e-2a401a986fb7-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:14:36.344726 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.344692    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4g8h\" (UniqueName: \"kubernetes.io/projected/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kube-api-access-d4g8h\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:14:36.344726 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.344707    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce58a5a-e1f7-47fd-946e-2a401a986fb7-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:14:36.344726 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:36.344723    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce58a5a-e1f7-47fd-946e-2a401a986fb7-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:14:37.023956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:37.023920    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5" event={"ID":"cce58a5a-e1f7-47fd-946e-2a401a986fb7","Type":"ContainerDied","Data":"9fd44a9a6985e4c66eca88de61e6ffc85b56bf65da5de303840ce66f34c12538"}
May 20 16:14:37.023956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:37.023964    2556 scope.go:117] "RemoveContainer" containerID="7a7e46206c170859638020e395dec390bd5b3fe3c78fe258bb666213b7af01b7"
May 20 16:14:37.024454 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:37.023932    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"
May 20 16:14:37.031852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:37.031624    2556 scope.go:117] "RemoveContainer" containerID="69d7b6fe34497392c9df4d0b024d35bd389073c9fbc5cfca9bb00a5554291b4a"
May 20 16:14:37.038378 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:37.038217    2556 scope.go:117] "RemoveContainer" containerID="a2c4f84cc6b8e628db50e97165bf4fc85eb50f1bbc36fd68726400afa6707170"
May 20 16:14:37.044932 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:37.044911    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"]
May 20 16:14:37.049238 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:37.049218    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-x57h5"]
May 20 16:14:37.890841 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:37.890808    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" path="/var/lib/kubelet/pods/cce58a5a-e1f7-47fd-946e-2a401a986fb7/volumes"
May 20 16:14:39.031822 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:39.031789    2556 generic.go:358] "Generic (PLEG): container finished" podID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerID="3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff" exitCode=0
May 20 16:14:39.031822 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:39.031826    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" event={"ID":"ccddb7ab-09a6-4bb6-986f-99bee04569f8","Type":"ContainerDied","Data":"3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff"}
May 20 16:14:40.035986 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:40.035951    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" event={"ID":"ccddb7ab-09a6-4bb6-986f-99bee04569f8","Type":"ContainerStarted","Data":"265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1"}
May 20 16:14:40.035986 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:40.035991    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" event={"ID":"ccddb7ab-09a6-4bb6-986f-99bee04569f8","Type":"ContainerStarted","Data":"114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114"}
May 20 16:14:40.036438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:40.036257    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:40.056703 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:40.056664    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podStartSLOduration=7.056652906 podStartE2EDuration="7.056652906s" podCreationTimestamp="2026-05-20 16:14:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:14:40.054784651 +0000 UTC m=+2522.746040579" watchObservedRunningTime="2026-05-20 16:14:40.056652906 +0000 UTC m=+2522.747908833"
May 20 16:14:41.038758 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:41.038726    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:41.039918 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:41.039894    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:14:42.042830 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:42.042782    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:14:47.046964 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:47.046930    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:14:47.047426 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:47.047280    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:14:57.047496 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:14:57.047454    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:15:07.048135 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:07.048069    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:15:17.047888 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:17.047847    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:15:27.048265 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:27.048226    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:15:37.047638 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:37.047598    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:15:47.048341 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:47.048305    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:15:53.852638 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.852603    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"]
May 20 16:15:53.853147 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.852889    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" containerID="cri-o://114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114" gracePeriod=30
May 20 16:15:53.853147 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.852955    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kube-rbac-proxy" containerID="cri-o://265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1" gracePeriod=30
May 20 16:15:53.907665 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907641    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"]
May 20 16:15:53.907869 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907858    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container"
May 20 16:15:53.907869 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907870    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container"
May 20 16:15:53.907965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907884    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kube-rbac-proxy"
May 20 16:15:53.907965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907889    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kube-rbac-proxy"
May 20 16:15:53.907965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907900    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="storage-initializer"
May 20 16:15:53.907965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907906    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="storage-initializer"
May 20 16:15:53.907965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907943    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kube-rbac-proxy"
May 20 16:15:53.907965 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.907950    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="cce58a5a-e1f7-47fd-946e-2a401a986fb7" containerName="kserve-container"
May 20 16:15:53.910678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.910663    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:53.913017 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.912996    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
May 20 16:15:53.913131 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.913001    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
May 20 16:15:53.921087 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.921066    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"]
May 20 16:15:53.997968 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.997941    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4d78d7-7737-4398-81c3-d16cad80164b-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:53.998066 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.997997    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4d78d7-7737-4398-81c3-d16cad80164b-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:53.998066 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.998014    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4d78d7-7737-4398-81c3-d16cad80164b-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:53.998176 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:53.998059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pk52\" (UniqueName: \"kubernetes.io/projected/5c4d78d7-7737-4398-81c3-d16cad80164b-kube-api-access-7pk52\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.098877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.098843    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7pk52\" (UniqueName: \"kubernetes.io/projected/5c4d78d7-7737-4398-81c3-d16cad80164b-kube-api-access-7pk52\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.099058 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.098889    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4d78d7-7737-4398-81c3-d16cad80164b-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.099058 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.098942    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4d78d7-7737-4398-81c3-d16cad80164b-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.099058 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.098957    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4d78d7-7737-4398-81c3-d16cad80164b-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.099386 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.099364    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4d78d7-7737-4398-81c3-d16cad80164b-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.099658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.099634    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4d78d7-7737-4398-81c3-d16cad80164b-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.101385 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.101367    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4d78d7-7737-4398-81c3-d16cad80164b-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.107935 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.107877    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pk52\" (UniqueName: \"kubernetes.io/projected/5c4d78d7-7737-4398-81c3-d16cad80164b-kube-api-access-7pk52\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kjtp2\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.220339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.220310    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:54.234338 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.234315    2556 generic.go:358] "Generic (PLEG): container finished" podID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerID="265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1" exitCode=2
May 20 16:15:54.234462 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.234385    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" event={"ID":"ccddb7ab-09a6-4bb6-986f-99bee04569f8","Type":"ContainerDied","Data":"265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1"}
May 20 16:15:54.342536 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.342515    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"]
May 20 16:15:54.344752 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:15:54.344713    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4d78d7_7737_4398_81c3_d16cad80164b.slice/crio-bd2c14982b60292324819565d9e0734f34a2006fd98949f51208934026c81e72 WatchSource:0}: Error finding container bd2c14982b60292324819565d9e0734f34a2006fd98949f51208934026c81e72: Status 404 returned error can't find the container with id bd2c14982b60292324819565d9e0734f34a2006fd98949f51208934026c81e72
May 20 16:15:54.346518 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:54.346502    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:15:55.238873 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:55.238837    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" event={"ID":"5c4d78d7-7737-4398-81c3-d16cad80164b","Type":"ContainerStarted","Data":"b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466"}
May 20 16:15:55.238873 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:55.238877    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" event={"ID":"5c4d78d7-7737-4398-81c3-d16cad80164b","Type":"ContainerStarted","Data":"bd2c14982b60292324819565d9e0734f34a2006fd98949f51208934026c81e72"}
May 20 16:15:57.043346 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.043260    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
May 20 16:15:57.047530 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.047509    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
May 20 16:15:57.696112 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.696084    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:15:57.725711 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.725688    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb7ab-09a6-4bb6-986f-99bee04569f8-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") "
May 20 16:15:57.725814 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.725727    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx5m7\" (UniqueName: \"kubernetes.io/projected/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kube-api-access-kx5m7\") pod \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") "
May 20 16:15:57.725865 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.725843    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb7ab-09a6-4bb6-986f-99bee04569f8-proxy-tls\") pod \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") "
May 20 16:15:57.725929 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.725902    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kserve-provision-location\") pod \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\" (UID: \"ccddb7ab-09a6-4bb6-986f-99bee04569f8\") "
May 20 16:15:57.726111 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.726071    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccddb7ab-09a6-4bb6-986f-99bee04569f8-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "ccddb7ab-09a6-4bb6-986f-99bee04569f8" (UID: "ccddb7ab-09a6-4bb6-986f-99bee04569f8"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:15:57.726191 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.726164    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb7ab-09a6-4bb6-986f-99bee04569f8-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:15:57.726252 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.726235    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ccddb7ab-09a6-4bb6-986f-99bee04569f8" (UID: "ccddb7ab-09a6-4bb6-986f-99bee04569f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:57.727797 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.727770    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kube-api-access-kx5m7" (OuterVolumeSpecName: "kube-api-access-kx5m7") pod "ccddb7ab-09a6-4bb6-986f-99bee04569f8" (UID: "ccddb7ab-09a6-4bb6-986f-99bee04569f8"). InnerVolumeSpecName "kube-api-access-kx5m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:15:57.727905 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.727857    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccddb7ab-09a6-4bb6-986f-99bee04569f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ccddb7ab-09a6-4bb6-986f-99bee04569f8" (UID: "ccddb7ab-09a6-4bb6-986f-99bee04569f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:15:57.827426 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.827366    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb7ab-09a6-4bb6-986f-99bee04569f8-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:15:57.827426 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.827390    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:15:57.827426 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:57.827404    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx5m7\" (UniqueName: \"kubernetes.io/projected/ccddb7ab-09a6-4bb6-986f-99bee04569f8-kube-api-access-kx5m7\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:15:58.248322 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.248294    2556 generic.go:358] "Generic (PLEG): container finished" podID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerID="114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114" exitCode=0
May 20 16:15:58.248658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.248382    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"
May 20 16:15:58.248658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.248376    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" event={"ID":"ccddb7ab-09a6-4bb6-986f-99bee04569f8","Type":"ContainerDied","Data":"114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114"}
May 20 16:15:58.248658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.248502    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l" event={"ID":"ccddb7ab-09a6-4bb6-986f-99bee04569f8","Type":"ContainerDied","Data":"666f2fc7826e8a55c2aede0c1ecf5069fc704c334d1bb28404c1eec1dd59534b"}
May 20 16:15:58.248658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.248523    2556 scope.go:117] "RemoveContainer" containerID="265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1"
May 20 16:15:58.249662 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.249641    2556 generic.go:358] "Generic (PLEG): container finished" podID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerID="b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466" exitCode=0
May 20 16:15:58.249738 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.249711    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" event={"ID":"5c4d78d7-7737-4398-81c3-d16cad80164b","Type":"ContainerDied","Data":"b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466"}
May 20 16:15:58.282079 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.282048    2556 scope.go:117] "RemoveContainer" containerID="114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114"
May 20 16:15:58.303139 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.303121    2556 scope.go:117] "RemoveContainer" containerID="3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff"
May 20 16:15:58.310135 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.310118    2556 scope.go:117] "RemoveContainer" containerID="265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1"
May 20 16:15:58.310386 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:15:58.310369    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1\": container with ID starting with 265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1 not found: ID does not exist" containerID="265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1"
May 20 16:15:58.310448 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.310393    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1"} err="failed to get container status \"265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1\": rpc error: code = NotFound desc = could not find container \"265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1\": container with ID starting with 265897a9bad50a669a9ae35b60aa9fd7ed16d91243b557455ce162b6589e48e1 not found: ID does not exist"
May 20 16:15:58.310448 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.310409    2556 scope.go:117] "RemoveContainer" containerID="114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114"
May 20 16:15:58.310618 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:15:58.310597    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114\": container with ID starting with 114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114 not found: ID does not exist" containerID="114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114"
May 20 16:15:58.310661 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.310625    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114"} err="failed to get container status \"114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114\": rpc error: code = NotFound desc = could not find container \"114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114\": container with ID starting with 114f82e5bf6cd4273d35ae2d308d5e005a16be05092df4aa26ae6791bee20114 not found: ID does not exist"
May 20 16:15:58.310661 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.310642    2556 scope.go:117] "RemoveContainer" containerID="3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff"
May 20 16:15:58.310865 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:15:58.310848    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff\": container with ID starting with 3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff not found: ID does not exist" containerID="3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff"
May 20 16:15:58.310927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.310871    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff"} err="failed to get container status \"3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff\": rpc error: code = NotFound desc = could not find container \"3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff\": container with ID starting with 3ae7ccb688c1d6789924397fc3ec775f01af2be29b82fe25ea910f2d5c2227ff not found: ID does not exist"
May 20 16:15:58.316348 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.316327    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"]
May 20 16:15:58.327177 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:58.323644    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6b7bccc865-rfs5l"]
May 20 16:15:59.255072 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:59.255036    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" event={"ID":"5c4d78d7-7737-4398-81c3-d16cad80164b","Type":"ContainerStarted","Data":"7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de"}
May 20 16:15:59.255072 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:59.255073    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" event={"ID":"5c4d78d7-7737-4398-81c3-d16cad80164b","Type":"ContainerStarted","Data":"4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f"}
May 20 16:15:59.255564 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:59.255307    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:15:59.274548 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:59.274491    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" podStartSLOduration=6.27447238 podStartE2EDuration="6.27447238s" podCreationTimestamp="2026-05-20 16:15:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:15:59.272567848 +0000 UTC m=+2601.963823776" watchObservedRunningTime="2026-05-20 16:15:59.27447238 +0000 UTC m=+2601.965728309"
May 20 16:15:59.890707 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:15:59.890677    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" path="/var/lib/kubelet/pods/ccddb7ab-09a6-4bb6-986f-99bee04569f8/volumes"
May 20 16:16:00.257470 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:00.257438    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:16:06.265135 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:06.265089    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:16:36.305272 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:36.305231    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 20 16:16:46.268503 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:46.268470    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:16:54.023825 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.023790    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"]
May 20 16:16:54.024413 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.024228    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kserve-container" containerID="cri-o://4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f" gracePeriod=30
May 20 16:16:54.024413 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.024271    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kube-rbac-proxy" containerID="cri-o://7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de" gracePeriod=30
May 20 16:16:54.112648 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.112612    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"]
May 20 16:16:54.112982 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.112964    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container"
May 20 16:16:54.113027 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.112988    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container"
May 20 16:16:54.113027 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.113005    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kube-rbac-proxy"
May 20 16:16:54.113027 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.113013    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kube-rbac-proxy"
May 20 16:16:54.113146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.113026    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="storage-initializer"
May 20 16:16:54.113146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.113034    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="storage-initializer"
May 20 16:16:54.113215 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.113145    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kserve-container"
May 20 16:16:54.113215 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.113161    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccddb7ab-09a6-4bb6-986f-99bee04569f8" containerName="kube-rbac-proxy"
May 20 16:16:54.116444 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.116425    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.118839 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.118819    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
May 20 16:16:54.118947 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.118838    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
May 20 16:16:54.123126 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.123082    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"]
May 20 16:16:54.241977 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.241935    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e578738-d135-4bdc-b3b2-f26c83911b31-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.241977 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.241976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk65p\" (UniqueName: \"kubernetes.io/projected/7e578738-d135-4bdc-b3b2-f26c83911b31-kube-api-access-tk65p\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.242221 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.242093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e578738-d135-4bdc-b3b2-f26c83911b31-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.242221 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.242173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e578738-d135-4bdc-b3b2-f26c83911b31-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.342756 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.342667    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e578738-d135-4bdc-b3b2-f26c83911b31-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.342756 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.342707    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk65p\" (UniqueName: \"kubernetes.io/projected/7e578738-d135-4bdc-b3b2-f26c83911b31-kube-api-access-tk65p\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.343002 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.342773    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e578738-d135-4bdc-b3b2-f26c83911b31-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.343002 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.342810    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e578738-d135-4bdc-b3b2-f26c83911b31-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.343192 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.343171    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e578738-d135-4bdc-b3b2-f26c83911b31-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.343357 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.343338    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e578738-d135-4bdc-b3b2-f26c83911b31-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.345171 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.345153    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e578738-d135-4bdc-b3b2-f26c83911b31-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.351652 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.351632    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk65p\" (UniqueName: \"kubernetes.io/projected/7e578738-d135-4bdc-b3b2-f26c83911b31-kube-api-access-tk65p\") pod \"isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.401987 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.401962    2556 generic.go:358] "Generic (PLEG): container finished" podID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerID="7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de" exitCode=2
May 20 16:16:54.402092 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.402034    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" event={"ID":"5c4d78d7-7737-4398-81c3-d16cad80164b","Type":"ContainerDied","Data":"7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de"}
May 20 16:16:54.428222 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.428193    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:16:54.552794 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:54.552772    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"]
May 20 16:16:54.555186 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:16:54.555158    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e578738_d135_4bdc_b3b2_f26c83911b31.slice/crio-2adc912bd6e4fd4b9df876d15c2ee595188efd8e0589c1dff8b1ef391a58c68e WatchSource:0}: Error finding container 2adc912bd6e4fd4b9df876d15c2ee595188efd8e0589c1dff8b1ef391a58c68e: Status 404 returned error can't find the container with id 2adc912bd6e4fd4b9df876d15c2ee595188efd8e0589c1dff8b1ef391a58c68e
May 20 16:16:55.406452 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:55.406411    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" event={"ID":"7e578738-d135-4bdc-b3b2-f26c83911b31","Type":"ContainerStarted","Data":"c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec"}
May 20 16:16:55.406849 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:55.406461    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" event={"ID":"7e578738-d135-4bdc-b3b2-f26c83911b31","Type":"ContainerStarted","Data":"2adc912bd6e4fd4b9df876d15c2ee595188efd8e0589c1dff8b1ef391a58c68e"}
May 20 16:16:56.260500 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:16:56.260453    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
May 20 16:17:00.572045 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.572021    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:17:00.691782 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.691714    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4d78d7-7737-4398-81c3-d16cad80164b-proxy-tls\") pod \"5c4d78d7-7737-4398-81c3-d16cad80164b\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") "
May 20 16:17:00.691900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.691788    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4d78d7-7737-4398-81c3-d16cad80164b-kserve-provision-location\") pod \"5c4d78d7-7737-4398-81c3-d16cad80164b\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") "
May 20 16:17:00.691900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.691834    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4d78d7-7737-4398-81c3-d16cad80164b-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"5c4d78d7-7737-4398-81c3-d16cad80164b\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") "
May 20 16:17:00.691900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.691863    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pk52\" (UniqueName: \"kubernetes.io/projected/5c4d78d7-7737-4398-81c3-d16cad80164b-kube-api-access-7pk52\") pod \"5c4d78d7-7737-4398-81c3-d16cad80164b\" (UID: \"5c4d78d7-7737-4398-81c3-d16cad80164b\") "
May 20 16:17:00.692175 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.692145    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4d78d7-7737-4398-81c3-d16cad80164b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c4d78d7-7737-4398-81c3-d16cad80164b" (UID: "5c4d78d7-7737-4398-81c3-d16cad80164b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:17:00.692252 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.692192    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4d78d7-7737-4398-81c3-d16cad80164b-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "5c4d78d7-7737-4398-81c3-d16cad80164b" (UID: "5c4d78d7-7737-4398-81c3-d16cad80164b"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:17:00.693636 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.693618    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4d78d7-7737-4398-81c3-d16cad80164b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c4d78d7-7737-4398-81c3-d16cad80164b" (UID: "5c4d78d7-7737-4398-81c3-d16cad80164b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:17:00.693920 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.693904    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4d78d7-7737-4398-81c3-d16cad80164b-kube-api-access-7pk52" (OuterVolumeSpecName: "kube-api-access-7pk52") pod "5c4d78d7-7737-4398-81c3-d16cad80164b" (UID: "5c4d78d7-7737-4398-81c3-d16cad80164b"). InnerVolumeSpecName "kube-api-access-7pk52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:17:00.792594 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.792553    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4d78d7-7737-4398-81c3-d16cad80164b-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:17:00.792594 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.792590    2556 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4d78d7-7737-4398-81c3-d16cad80164b-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:17:00.792594 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.792602    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pk52\" (UniqueName: \"kubernetes.io/projected/5c4d78d7-7737-4398-81c3-d16cad80164b-kube-api-access-7pk52\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:17:00.792813 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:00.792613    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4d78d7-7737-4398-81c3-d16cad80164b-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:17:01.425634 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.425599    2556 generic.go:358] "Generic (PLEG): container finished" podID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerID="4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f" exitCode=0
May 20 16:17:01.425814 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.425674    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" event={"ID":"5c4d78d7-7737-4398-81c3-d16cad80164b","Type":"ContainerDied","Data":"4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f"}
May 20 16:17:01.425814 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.425699    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"
May 20 16:17:01.425814 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.425718    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2" event={"ID":"5c4d78d7-7737-4398-81c3-d16cad80164b","Type":"ContainerDied","Data":"bd2c14982b60292324819565d9e0734f34a2006fd98949f51208934026c81e72"}
May 20 16:17:01.425814 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.425736    2556 scope.go:117] "RemoveContainer" containerID="7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de"
May 20 16:17:01.427305 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.427279    2556 generic.go:358] "Generic (PLEG): container finished" podID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerID="c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec" exitCode=0
May 20 16:17:01.427447 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.427315    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" event={"ID":"7e578738-d135-4bdc-b3b2-f26c83911b31","Type":"ContainerDied","Data":"c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec"}
May 20 16:17:01.435133 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.435112    2556 scope.go:117] "RemoveContainer" containerID="4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f"
May 20 16:17:01.442782 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.442631    2556 scope.go:117] "RemoveContainer" containerID="b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466"
May 20 16:17:01.453453 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.453426    2556 scope.go:117] "RemoveContainer" containerID="7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de"
May 20 16:17:01.453720 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:17:01.453702    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de\": container with ID starting with 7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de not found: ID does not exist" containerID="7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de"
May 20 16:17:01.453770 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.453727    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de"} err="failed to get container status \"7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de\": rpc error: code = NotFound desc = could not find container \"7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de\": container with ID starting with 7e1f606e615dc55bc0170d0ac29b37100aba268f2409afdab017c08834f303de not found: ID does not exist"
May 20 16:17:01.453770 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.453744    2556 scope.go:117] "RemoveContainer" containerID="4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f"
May 20 16:17:01.454005 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:17:01.453986    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f\": container with ID starting with 4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f not found: ID does not exist" containerID="4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f"
May 20 16:17:01.454176 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.454010    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f"} err="failed to get container status \"4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f\": rpc error: code = NotFound desc = could not find container \"4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f\": container with ID starting with 4c735fd345ea5eb3c21e78b4f7daff83531e6cc0f9e4a93c7e3359febdf05c4f not found: ID does not exist"
May 20 16:17:01.454176 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.454028    2556 scope.go:117] "RemoveContainer" containerID="b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466"
May 20 16:17:01.454281 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:17:01.454267    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466\": container with ID starting with b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466 not found: ID does not exist" containerID="b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466"
May 20 16:17:01.454320 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.454283    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466"} err="failed to get container status \"b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466\": rpc error: code = NotFound desc = could not find container \"b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466\": container with ID starting with b0508d65e00b1c702008fe5a90373aa0195106979623aab192e8ed8d7508e466 not found: ID does not exist"
May 20 16:17:01.460466 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.460446    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"]
May 20 16:17:01.465270 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.465245    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kjtp2"]
May 20 16:17:01.890503 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:01.890463    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" path="/var/lib/kubelet/pods/5c4d78d7-7737-4398-81c3-d16cad80164b/volumes"
May 20 16:17:02.432516 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:02.432487    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" event={"ID":"7e578738-d135-4bdc-b3b2-f26c83911b31","Type":"ContainerStarted","Data":"283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1"}
May 20 16:17:02.432516 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:02.432522    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" event={"ID":"7e578738-d135-4bdc-b3b2-f26c83911b31","Type":"ContainerStarted","Data":"39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa"}
May 20 16:17:02.432821 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:02.432800    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:17:02.432935 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:02.432923    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:17:02.434155 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:02.434131    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 20 16:17:02.451588 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:02.451551    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" podStartSLOduration=8.451537388 podStartE2EDuration="8.451537388s" podCreationTimestamp="2026-05-20 16:16:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:17:02.450027547 +0000 UTC m=+2665.141283475" watchObservedRunningTime="2026-05-20 16:17:02.451537388 +0000 UTC m=+2665.142793315"
May 20 16:17:03.434988 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:03.434945    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 20 16:17:08.439204 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:08.439175    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:17:08.439760 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:08.439733    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
May 20 16:17:18.440310 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:18.440278    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:17:31.134788 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.134690    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp_7e578738-d135-4bdc-b3b2-f26c83911b31/kserve-container/0.log"
May 20 16:17:31.290720 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.290686    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"]
May 20 16:17:31.291092 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.291055    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kserve-container" containerID="cri-o://39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa" gracePeriod=30
May 20 16:17:31.291289 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.291078    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kube-rbac-proxy" containerID="cri-o://283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1" gracePeriod=30
May 20 16:17:31.354230 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354198    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"]
May 20 16:17:31.354455 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354443    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="storage-initializer"
May 20 16:17:31.354498 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354456    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="storage-initializer"
May 20 16:17:31.354498 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354467    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kserve-container"
May 20 16:17:31.354498 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354472    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kserve-container"
May 20 16:17:31.354498 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354485    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kube-rbac-proxy"
May 20 16:17:31.354498 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354491    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kube-rbac-proxy"
May 20 16:17:31.354651 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354532    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kserve-container"
May 20 16:17:31.354651 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.354541    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c4d78d7-7737-4398-81c3-d16cad80164b" containerName="kube-rbac-proxy"
May 20 16:17:31.356514 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.356499    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.358948 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.358928    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
May 20 16:17:31.359060 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.358929    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
May 20 16:17:31.366891 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.366869    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"]
May 20 16:17:31.413073 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.413055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.413155 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.413083    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kube-api-access-sgzfj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.413155 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.413130    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.413237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.413163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.511501 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.511477    2556 generic.go:358] "Generic (PLEG): container finished" podID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerID="283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1" exitCode=2
May 20 16:17:31.511598 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.511520    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" event={"ID":"7e578738-d135-4bdc-b3b2-f26c83911b31","Type":"ContainerDied","Data":"283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1"}
May 20 16:17:31.513782 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.513764    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.513834 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.513806    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.513834 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.513823    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kube-api-access-sgzfj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.513971 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.513946    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.514218 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.514199    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.514533 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.514518    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.515910 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.515892    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.522607 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.522587    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kube-api-access-sgzfj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.666956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.666856    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:31.794833 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:31.794800    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"]
May 20 16:17:31.798189 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:17:31.798160    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5e4b74_1afe_4a98_9e70_ef7bc9fedeac.slice/crio-11a6d7e89750d9a59022039e81de86b1fae7eb3ac5c9662843d9cf18c1943b0a WatchSource:0}: Error finding container 11a6d7e89750d9a59022039e81de86b1fae7eb3ac5c9662843d9cf18c1943b0a: Status 404 returned error can't find the container with id 11a6d7e89750d9a59022039e81de86b1fae7eb3ac5c9662843d9cf18c1943b0a
May 20 16:17:32.427933 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.427912    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:17:32.515412 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.515341    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" event={"ID":"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac","Type":"ContainerStarted","Data":"a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd"}
May 20 16:17:32.515412 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.515374    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" event={"ID":"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac","Type":"ContainerStarted","Data":"11a6d7e89750d9a59022039e81de86b1fae7eb3ac5c9662843d9cf18c1943b0a"}
May 20 16:17:32.516900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.516873    2556 generic.go:358] "Generic (PLEG): container finished" podID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerID="39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa" exitCode=0
May 20 16:17:32.517012 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.516904    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" event={"ID":"7e578738-d135-4bdc-b3b2-f26c83911b31","Type":"ContainerDied","Data":"39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa"}
May 20 16:17:32.517012 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.516926    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"
May 20 16:17:32.517012 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.516936    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp" event={"ID":"7e578738-d135-4bdc-b3b2-f26c83911b31","Type":"ContainerDied","Data":"2adc912bd6e4fd4b9df876d15c2ee595188efd8e0589c1dff8b1ef391a58c68e"}
May 20 16:17:32.517012 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.516952    2556 scope.go:117] "RemoveContainer" containerID="283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1"
May 20 16:17:32.519916 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.519895    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk65p\" (UniqueName: \"kubernetes.io/projected/7e578738-d135-4bdc-b3b2-f26c83911b31-kube-api-access-tk65p\") pod \"7e578738-d135-4bdc-b3b2-f26c83911b31\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") "
May 20 16:17:32.520031 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.519938    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e578738-d135-4bdc-b3b2-f26c83911b31-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"7e578738-d135-4bdc-b3b2-f26c83911b31\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") "
May 20 16:17:32.520031 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.519960    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e578738-d135-4bdc-b3b2-f26c83911b31-kserve-provision-location\") pod \"7e578738-d135-4bdc-b3b2-f26c83911b31\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") "
May 20 16:17:32.520031 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.520022    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e578738-d135-4bdc-b3b2-f26c83911b31-proxy-tls\") pod \"7e578738-d135-4bdc-b3b2-f26c83911b31\" (UID: \"7e578738-d135-4bdc-b3b2-f26c83911b31\") "
May 20 16:17:32.520401 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.520376    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e578738-d135-4bdc-b3b2-f26c83911b31-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "7e578738-d135-4bdc-b3b2-f26c83911b31" (UID: "7e578738-d135-4bdc-b3b2-f26c83911b31"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:17:32.522152 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.522127    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e578738-d135-4bdc-b3b2-f26c83911b31-kube-api-access-tk65p" (OuterVolumeSpecName: "kube-api-access-tk65p") pod "7e578738-d135-4bdc-b3b2-f26c83911b31" (UID: "7e578738-d135-4bdc-b3b2-f26c83911b31"). InnerVolumeSpecName "kube-api-access-tk65p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:17:32.522259 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.522199    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e578738-d135-4bdc-b3b2-f26c83911b31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e578738-d135-4bdc-b3b2-f26c83911b31" (UID: "7e578738-d135-4bdc-b3b2-f26c83911b31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:17:32.525142 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.525126    2556 scope.go:117] "RemoveContainer" containerID="39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa"
May 20 16:17:32.534840 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.534818    2556 scope.go:117] "RemoveContainer" containerID="c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec"
May 20 16:17:32.541360 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.541344    2556 scope.go:117] "RemoveContainer" containerID="283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1"
May 20 16:17:32.541622 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:17:32.541602    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1\": container with ID starting with 283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1 not found: ID does not exist" containerID="283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1"
May 20 16:17:32.541670 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.541629    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1"} err="failed to get container status \"283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1\": rpc error: code = NotFound desc = could not find container \"283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1\": container with ID starting with 283297e2556a5ff5168ceabfe10f9b8134564047ade42feab1ae21855dab0fe1 not found: ID does not exist"
May 20 16:17:32.541670 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.541646    2556 scope.go:117] "RemoveContainer" containerID="39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa"
May 20 16:17:32.541888 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:17:32.541873    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa\": container with ID starting with 39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa not found: ID does not exist" containerID="39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa"
May 20 16:17:32.541938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.541893    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa"} err="failed to get container status \"39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa\": rpc error: code = NotFound desc = could not find container \"39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa\": container with ID starting with 39b58397452a6ccc46249d98c8a8bfb642525eea4f73146dc620612c3fe919fa not found: ID does not exist"
May 20 16:17:32.541938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.541907    2556 scope.go:117] "RemoveContainer" containerID="c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec"
May 20 16:17:32.542159 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:17:32.542135    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec\": container with ID starting with c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec not found: ID does not exist" containerID="c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec"
May 20 16:17:32.542203 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.542167    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec"} err="failed to get container status \"c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec\": rpc error: code = NotFound desc = could not find container \"c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec\": container with ID starting with c9d54a5397cf16b0e0bd8a83acecf82ce35d89e62f9b6614cd6b9f29205a71ec not found: ID does not exist"
May 20 16:17:32.545323 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.545304    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e578738-d135-4bdc-b3b2-f26c83911b31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e578738-d135-4bdc-b3b2-f26c83911b31" (UID: "7e578738-d135-4bdc-b3b2-f26c83911b31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:17:32.621472 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.621441    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk65p\" (UniqueName: \"kubernetes.io/projected/7e578738-d135-4bdc-b3b2-f26c83911b31-kube-api-access-tk65p\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:17:32.621472 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.621473    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e578738-d135-4bdc-b3b2-f26c83911b31-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:17:32.621605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.621485    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e578738-d135-4bdc-b3b2-f26c83911b31-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:17:32.621605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.621494    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e578738-d135-4bdc-b3b2-f26c83911b31-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:17:32.841273 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.841243    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"]
May 20 16:17:32.845179 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:32.845157    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-fbf45cfb7-pctcp"]
May 20 16:17:33.890618 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:33.890585    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" path="/var/lib/kubelet/pods/7e578738-d135-4bdc-b3b2-f26c83911b31/volumes"
May 20 16:17:36.530735 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:36.530700    2556 generic.go:358] "Generic (PLEG): container finished" podID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerID="a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd" exitCode=0
May 20 16:17:36.531137 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:36.530773    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" event={"ID":"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac","Type":"ContainerDied","Data":"a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd"}
May 20 16:17:37.535718 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:37.535681    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" event={"ID":"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac","Type":"ContainerStarted","Data":"9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc"}
May 20 16:17:37.535718 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:37.535720    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" event={"ID":"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac","Type":"ContainerStarted","Data":"519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504"}
May 20 16:17:37.536176 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:37.535934    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:37.536176 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:37.535968    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:43.544297 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:43.544270    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:17:43.564777 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:17:43.564735    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" podStartSLOduration=12.564722181 podStartE2EDuration="12.564722181s" podCreationTimestamp="2026-05-20 16:17:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:17:37.553947686 +0000 UTC m=+2700.245203614" watchObservedRunningTime="2026-05-20 16:17:43.564722181 +0000 UTC m=+2706.255978092"
May 20 16:18:13.604957 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:13.604907    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 20 16:18:23.547267 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:23.547236    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:18:31.491727 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.491690    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"]
May 20 16:18:31.492177 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.492025    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kserve-container" containerID="cri-o://519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504" gracePeriod=30
May 20 16:18:31.492177 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.492078    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kube-rbac-proxy" containerID="cri-o://9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc" gracePeriod=30
May 20 16:18:31.550191 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550162    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"]
May 20 16:18:31.550428 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550416    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kube-rbac-proxy"
May 20 16:18:31.550476 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550430    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kube-rbac-proxy"
May 20 16:18:31.550476 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550438    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="storage-initializer"
May 20 16:18:31.550476 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550445    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="storage-initializer"
May 20 16:18:31.550476 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550454    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kserve-container"
May 20 16:18:31.550476 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550459    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kserve-container"
May 20 16:18:31.550630 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550498    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kube-rbac-proxy"
May 20 16:18:31.550630 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.550508    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="7e578738-d135-4bdc-b3b2-f26c83911b31" containerName="kserve-container"
May 20 16:18:31.552423 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.552409    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.554837 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.554815    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
May 20 16:18:31.554930 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.554844    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
May 20 16:18:31.563980 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.563958    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"]
May 20 16:18:31.629218 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.629194    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.629297 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.629228    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.629297 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.629248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-proxy-tls\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.629402 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.629324    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qw78\" (UniqueName: \"kubernetes.io/projected/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kube-api-access-7qw78\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.682045 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.682023    2556 generic.go:358] "Generic (PLEG): container finished" podID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerID="9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc" exitCode=2
May 20 16:18:31.682158 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.682056    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" event={"ID":"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac","Type":"ContainerDied","Data":"9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc"}
May 20 16:18:31.730614 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.730593    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7qw78\" (UniqueName: \"kubernetes.io/projected/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kube-api-access-7qw78\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.730701 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.730645    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.730701 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.730664    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.730701 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.730694    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-proxy-tls\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.730995 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.730975    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.731343 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.731323    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.732873 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.732857    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-proxy-tls\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.739488 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.739467    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qw78\" (UniqueName: \"kubernetes.io/projected/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kube-api-access-7qw78\") pod \"isvc-sklearn-v2-predictor-79646f46b9-dmgqt\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.861653 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.861602    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:31.980797 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:31.980772    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"]
May 20 16:18:31.982955 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:18:31.982921    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b63c2e_372d_41d1_9e59_3c5ec7eaacae.slice/crio-8de66d52914308ecf2e7e8cb54a7ed0bb0352a913ce339cf6f987c4728eb60e9 WatchSource:0}: Error finding container 8de66d52914308ecf2e7e8cb54a7ed0bb0352a913ce339cf6f987c4728eb60e9: Status 404 returned error can't find the container with id 8de66d52914308ecf2e7e8cb54a7ed0bb0352a913ce339cf6f987c4728eb60e9
May 20 16:18:32.686683 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:32.686645    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" event={"ID":"85b63c2e-372d-41d1-9e59-3c5ec7eaacae","Type":"ContainerStarted","Data":"0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce"}
May 20 16:18:32.686683 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:32.686684    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" event={"ID":"85b63c2e-372d-41d1-9e59-3c5ec7eaacae","Type":"ContainerStarted","Data":"8de66d52914308ecf2e7e8cb54a7ed0bb0352a913ce339cf6f987c4728eb60e9"}
May 20 16:18:33.539038 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:33.539000    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
May 20 16:18:36.698605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:36.698568    2556 generic.go:358] "Generic (PLEG): container finished" podID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerID="0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce" exitCode=0
May 20 16:18:36.699167 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:36.698621    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" event={"ID":"85b63c2e-372d-41d1-9e59-3c5ec7eaacae","Type":"ContainerDied","Data":"0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce"}
May 20 16:18:37.702949 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.702925    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" event={"ID":"85b63c2e-372d-41d1-9e59-3c5ec7eaacae","Type":"ContainerStarted","Data":"f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b"}
May 20 16:18:37.703368 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.702956    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" event={"ID":"85b63c2e-372d-41d1-9e59-3c5ec7eaacae","Type":"ContainerStarted","Data":"3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54"}
May 20 16:18:37.703368 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.703184    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:37.724664 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.724622    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podStartSLOduration=6.724608423 podStartE2EDuration="6.724608423s" podCreationTimestamp="2026-05-20 16:18:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:18:37.722637508 +0000 UTC m=+2760.413893435" watchObservedRunningTime="2026-05-20 16:18:37.724608423 +0000 UTC m=+2760.415864422"
May 20 16:18:37.827419 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.827397    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:18:37.871858 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.871807    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-proxy-tls\") pod \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") "
May 20 16:18:37.871858 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.871849    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kube-api-access-sgzfj\") pod \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") "
May 20 16:18:37.871991 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.871872    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kserve-provision-location\") pod \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") "
May 20 16:18:37.871991 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.871923    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\" (UID: \"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac\") "
May 20 16:18:37.872227 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.872201    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" (UID: "fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:18:37.872308 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.872285    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" (UID: "fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:18:37.873828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.873805    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kube-api-access-sgzfj" (OuterVolumeSpecName: "kube-api-access-sgzfj") pod "fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" (UID: "fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac"). InnerVolumeSpecName "kube-api-access-sgzfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:18:37.873828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.873822    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" (UID: "fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:18:37.972794 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.972770    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:18:37.972794 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.972792    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzfj\" (UniqueName: \"kubernetes.io/projected/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kube-api-access-sgzfj\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:18:37.972932 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.972802    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:18:37.972932 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:37.972814    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:18:38.707776 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.707738    2556 generic.go:358] "Generic (PLEG): container finished" podID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerID="519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504" exitCode=0
May 20 16:18:38.708299 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.707825    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"
May 20 16:18:38.708299 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.707833    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" event={"ID":"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac","Type":"ContainerDied","Data":"519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504"}
May 20 16:18:38.708299 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.707887    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x" event={"ID":"fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac","Type":"ContainerDied","Data":"11a6d7e89750d9a59022039e81de86b1fae7eb3ac5c9662843d9cf18c1943b0a"}
May 20 16:18:38.708299 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.707912    2556 scope.go:117] "RemoveContainer" containerID="9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc"
May 20 16:18:38.708541 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.708393    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:38.710172 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.710143    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:18:38.716231 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.716214    2556 scope.go:117] "RemoveContainer" containerID="519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504"
May 20 16:18:38.722826 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.722809    2556 scope.go:117] "RemoveContainer" containerID="a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd"
May 20 16:18:38.725752 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.725727    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"]
May 20 16:18:38.729686 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.729663    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-ctl2x"]
May 20 16:18:38.730747 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.730731    2556 scope.go:117] "RemoveContainer" containerID="9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc"
May 20 16:18:38.730998 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:18:38.730980    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc\": container with ID starting with 9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc not found: ID does not exist" containerID="9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc"
May 20 16:18:38.731057 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.731005    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc"} err="failed to get container status \"9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc\": rpc error: code = NotFound desc = could not find container \"9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc\": container with ID starting with 9b57aa397aed953104b7c025239a9f43add39089b3def233bf08f56a1439a0bc not found: ID does not exist"
May 20 16:18:38.731057 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.731023    2556 scope.go:117] "RemoveContainer" containerID="519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504"
May 20 16:18:38.731298 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:18:38.731282    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504\": container with ID starting with 519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504 not found: ID does not exist" containerID="519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504"
May 20 16:18:38.731348 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.731303    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504"} err="failed to get container status \"519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504\": rpc error: code = NotFound desc = could not find container \"519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504\": container with ID starting with 519a0714bd66778199b8f2b73d861dbe0659139e2c5b3e95d7a08786f417d504 not found: ID does not exist"
May 20 16:18:38.731348 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.731318    2556 scope.go:117] "RemoveContainer" containerID="a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd"
May 20 16:18:38.731577 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:18:38.731559    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd\": container with ID starting with a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd not found: ID does not exist" containerID="a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd"
May 20 16:18:38.731622 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:38.731583    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd"} err="failed to get container status \"a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd\": rpc error: code = NotFound desc = could not find container \"a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd\": container with ID starting with a716e5c1a7138b4951b2dc4f9fccdd6623decf742b056f97c851d5ee1aa813fd not found: ID does not exist"
May 20 16:18:39.711352 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:39.711270    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:18:39.891561 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:39.891527    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" path="/var/lib/kubelet/pods/fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac/volumes"
May 20 16:18:44.717120 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:44.717070    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:18:44.717657 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:44.717630    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:18:54.718526 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:18:54.718484    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:19:04.717959 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:04.717920    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:19:14.718084 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:14.718044    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:19:24.718271 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:24.718230    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:19:34.717741 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:34.717702    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:19:44.719112 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:44.719068    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:19:51.779848 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.779803    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"]
May 20 16:19:51.780330 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.780148    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" containerID="cri-o://3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54" gracePeriod=30
May 20 16:19:51.780330 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.780208    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kube-rbac-proxy" containerID="cri-o://f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b" gracePeriod=30
May 20 16:19:51.839973 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.839944    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"]
May 20 16:19:51.840222 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.840208    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="storage-initializer"
May 20 16:19:51.840273 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.840223    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="storage-initializer"
May 20 16:19:51.840273 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.840231    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kserve-container"
May 20 16:19:51.840273 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.840237    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kserve-container"
May 20 16:19:51.840273 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.840248    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kube-rbac-proxy"
May 20 16:19:51.840273 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.840255    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kube-rbac-proxy"
May 20 16:19:51.840433 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.840301    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kserve-container"
May 20 16:19:51.840433 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.840311    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa5e4b74-1afe-4a98-9e70-ef7bc9fedeac" containerName="kube-rbac-proxy"
May 20 16:19:51.842226 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.842211    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.844525 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.844506    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
May 20 16:19:51.844760 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.844740    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
May 20 16:19:51.852740 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.852721    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"]
May 20 16:19:51.879373 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.879344    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46sjl\" (UniqueName: \"kubernetes.io/projected/54ba2153-f238-430f-bd15-ac82c13570cc-kube-api-access-46sjl\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.879478 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.879395    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54ba2153-f238-430f-bd15-ac82c13570cc-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.879478 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.879429    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54ba2153-f238-430f-bd15-ac82c13570cc-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.879478 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.879456    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.908124 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.908085    2556 generic.go:358] "Generic (PLEG): container finished" podID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerID="f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b" exitCode=2
May 20 16:19:51.908217 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.908125    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" event={"ID":"85b63c2e-372d-41d1-9e59-3c5ec7eaacae","Type":"ContainerDied","Data":"f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b"}
May 20 16:19:51.980275 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.980236    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54ba2153-f238-430f-bd15-ac82c13570cc-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.980440 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.980295    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54ba2153-f238-430f-bd15-ac82c13570cc-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.980440 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.980334    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.980440 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.980391    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-46sjl\" (UniqueName: \"kubernetes.io/projected/54ba2153-f238-430f-bd15-ac82c13570cc-kube-api-access-46sjl\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.980615 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:19:51.980503    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
May 20 16:19:51.980615 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:19:51.980596    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls podName:54ba2153-f238-430f-bd15-ac82c13570cc nodeName:}" failed. No retries permitted until 2026-05-20 16:19:52.48057366 +0000 UTC m=+2835.171829576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" (UID: "54ba2153-f238-430f-bd15-ac82c13570cc") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
May 20 16:19:51.980740 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.980723    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54ba2153-f238-430f-bd15-ac82c13570cc-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.980989 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.980970    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54ba2153-f238-430f-bd15-ac82c13570cc-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:51.991283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:51.991261    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46sjl\" (UniqueName: \"kubernetes.io/projected/54ba2153-f238-430f-bd15-ac82c13570cc-kube-api-access-46sjl\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:52.484211 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:52.484177    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:52.486563 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:52.486540    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:52.752407 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:52.752332    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:52.875370 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:52.875341    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"]
May 20 16:19:52.912131 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:52.912085    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" event={"ID":"54ba2153-f238-430f-bd15-ac82c13570cc","Type":"ContainerStarted","Data":"d49c81ceb85d2f8e8a76dcaba297ed121e1f54023d41a6f6ee8735cc69a307b1"}
May 20 16:19:53.915824 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:53.915787    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" event={"ID":"54ba2153-f238-430f-bd15-ac82c13570cc","Type":"ContainerStarted","Data":"2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1"}
May 20 16:19:54.711851 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:54.711802    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
May 20 16:19:54.717969 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:54.717945    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
May 20 16:19:55.614354 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.614328    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:19:55.710149 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.710057    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-proxy-tls\") pod \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") "
May 20 16:19:55.710149 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.710145    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") "
May 20 16:19:55.710342 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.710168    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qw78\" (UniqueName: \"kubernetes.io/projected/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kube-api-access-7qw78\") pod \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") "
May 20 16:19:55.710342 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.710201    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kserve-provision-location\") pod \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\" (UID: \"85b63c2e-372d-41d1-9e59-3c5ec7eaacae\") "
May 20 16:19:55.710507 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.710488    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "85b63c2e-372d-41d1-9e59-3c5ec7eaacae" (UID: "85b63c2e-372d-41d1-9e59-3c5ec7eaacae"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:19:55.710589 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.710564    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85b63c2e-372d-41d1-9e59-3c5ec7eaacae" (UID: "85b63c2e-372d-41d1-9e59-3c5ec7eaacae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:19:55.711958 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.711937    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85b63c2e-372d-41d1-9e59-3c5ec7eaacae" (UID: "85b63c2e-372d-41d1-9e59-3c5ec7eaacae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:19:55.712074 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.712056    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kube-api-access-7qw78" (OuterVolumeSpecName: "kube-api-access-7qw78") pod "85b63c2e-372d-41d1-9e59-3c5ec7eaacae" (UID: "85b63c2e-372d-41d1-9e59-3c5ec7eaacae"). InnerVolumeSpecName "kube-api-access-7qw78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:19:55.811585 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.811554    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:19:55.811585 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.811586    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:19:55.811712 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.811598    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qw78\" (UniqueName: \"kubernetes.io/projected/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kube-api-access-7qw78\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:19:55.811712 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.811608    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b63c2e-372d-41d1-9e59-3c5ec7eaacae-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:19:55.922661 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.922634    2556 generic.go:358] "Generic (PLEG): container finished" podID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerID="3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54" exitCode=0
May 20 16:19:55.922760 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.922669    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" event={"ID":"85b63c2e-372d-41d1-9e59-3c5ec7eaacae","Type":"ContainerDied","Data":"3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54"}
May 20 16:19:55.922760 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.922689    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt" event={"ID":"85b63c2e-372d-41d1-9e59-3c5ec7eaacae","Type":"ContainerDied","Data":"8de66d52914308ecf2e7e8cb54a7ed0bb0352a913ce339cf6f987c4728eb60e9"}
May 20 16:19:55.922760 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.922710    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"
May 20 16:19:55.922865 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.922710    2556 scope.go:117] "RemoveContainer" containerID="f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b"
May 20 16:19:55.930255 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.930238    2556 scope.go:117] "RemoveContainer" containerID="3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54"
May 20 16:19:55.936474 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.936456    2556 scope.go:117] "RemoveContainer" containerID="0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce"
May 20 16:19:55.939913 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.939890    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"]
May 20 16:19:55.943270 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.943231    2556 scope.go:117] "RemoveContainer" containerID="f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b"
May 20 16:19:55.943526 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:19:55.943507    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b\": container with ID starting with f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b not found: ID does not exist" containerID="f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b"
May 20 16:19:55.943597 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.943533    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b"} err="failed to get container status \"f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b\": rpc error: code = NotFound desc = could not find container \"f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b\": container with ID starting with f1282c3204628ffc27f053a01ccece24204202750acd8bf8b78a5bdf66b39e6b not found: ID does not exist"
May 20 16:19:55.943597 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.943549    2556 scope.go:117] "RemoveContainer" containerID="3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54"
May 20 16:19:55.943826 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:19:55.943792    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54\": container with ID starting with 3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54 not found: ID does not exist" containerID="3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54"
May 20 16:19:55.943893 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.943823    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54"} err="failed to get container status \"3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54\": rpc error: code = NotFound desc = could not find container \"3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54\": container with ID starting with 3c72f03b7435ee60fafffd6453b89ea66960ac0412f1fed9e219ecd9b0eacd54 not found: ID does not exist"
May 20 16:19:55.943893 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.943838    2556 scope.go:117] "RemoveContainer" containerID="0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce"
May 20 16:19:55.944079 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:19:55.944063    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce\": container with ID starting with 0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce not found: ID does not exist" containerID="0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce"
May 20 16:19:55.944154 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.944084    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce"} err="failed to get container status \"0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce\": rpc error: code = NotFound desc = could not find container \"0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce\": container with ID starting with 0d2d6679ad88773e19d8b0c0228691a81407ff3997b3ae3a47fc08eede180bce not found: ID does not exist"
May 20 16:19:55.944408 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:55.944393    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-79646f46b9-dmgqt"]
May 20 16:19:56.927287 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:56.927211    2556 generic.go:358] "Generic (PLEG): container finished" podID="54ba2153-f238-430f-bd15-ac82c13570cc" containerID="2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1" exitCode=0
May 20 16:19:56.927702 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:56.927282    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" event={"ID":"54ba2153-f238-430f-bd15-ac82c13570cc","Type":"ContainerDied","Data":"2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1"}
May 20 16:19:57.890401 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:57.890368    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" path="/var/lib/kubelet/pods/85b63c2e-372d-41d1-9e59-3c5ec7eaacae/volumes"
May 20 16:19:57.931365 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:57.931339    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" event={"ID":"54ba2153-f238-430f-bd15-ac82c13570cc","Type":"ContainerStarted","Data":"03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18"}
May 20 16:19:57.931699 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:57.931370    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" event={"ID":"54ba2153-f238-430f-bd15-ac82c13570cc","Type":"ContainerStarted","Data":"313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe"}
May 20 16:19:57.931699 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:57.931646    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:57.931787 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:57.931751    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:19:57.933024 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:57.933004    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:19:57.950071 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:57.950026    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podStartSLOduration=6.950014926 podStartE2EDuration="6.950014926s" podCreationTimestamp="2026-05-20 16:19:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:19:57.948371038 +0000 UTC m=+2840.639626970" watchObservedRunningTime="2026-05-20 16:19:57.950014926 +0000 UTC m=+2840.641270928"
May 20 16:19:58.933825 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:19:58.933783    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:20:03.937905 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:20:03.937877    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:20:03.938467 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:20:03.938438    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:20:13.938529 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:20:13.938493    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:20:23.938468 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:20:23.938435    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:20:33.938944 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:20:33.938866    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:20:43.939053 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:20:43.939014    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:20:53.938834 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:20:53.938799    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:21:03.938914 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:03.938884    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:21:11.976806 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:11.976767    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"]
May 20 16:21:11.977292 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:11.977263    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" containerID="cri-o://313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe" gracePeriod=30
May 20 16:21:11.977383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:11.977265    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kube-rbac-proxy" containerID="cri-o://03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18" gracePeriod=30
May 20 16:21:12.059621 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059592    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"]
May 20 16:21:12.059834 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059821    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container"
May 20 16:21:12.059834 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059835    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container"
May 20 16:21:12.059927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059845    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kube-rbac-proxy"
May 20 16:21:12.059927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059851    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kube-rbac-proxy"
May 20 16:21:12.059927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059864    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="storage-initializer"
May 20 16:21:12.059927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059870    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="storage-initializer"
May 20 16:21:12.059927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059910    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kube-rbac-proxy"
May 20 16:21:12.059927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.059920    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="85b63c2e-372d-41d1-9e59-3c5ec7eaacae" containerName="kserve-container"
May 20 16:21:12.062990 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.062975    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.065436 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.065420    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
May 20 16:21:12.065518 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.065438    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
May 20 16:21:12.072958 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.072937    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"]
May 20 16:21:12.131351 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.131326    2556 generic.go:358] "Generic (PLEG): container finished" podID="54ba2153-f238-430f-bd15-ac82c13570cc" containerID="03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18" exitCode=2
May 20 16:21:12.131452 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.131404    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" event={"ID":"54ba2153-f238-430f-bd15-ac82c13570cc","Type":"ContainerDied","Data":"03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18"}
May 20 16:21:12.142855 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.142820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c89679-3b93-4575-a19f-c2300f30f78e-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.142944 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.142864    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c89679-3b93-4575-a19f-c2300f30f78e-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.142944 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.142881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bfl\" (UniqueName: \"kubernetes.io/projected/35c89679-3b93-4575-a19f-c2300f30f78e-kube-api-access-n5bfl\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.143024 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.142944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c89679-3b93-4575-a19f-c2300f30f78e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.244144 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.244121    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c89679-3b93-4575-a19f-c2300f30f78e-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.244246 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.244152    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c89679-3b93-4575-a19f-c2300f30f78e-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.244246 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.244168    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bfl\" (UniqueName: \"kubernetes.io/projected/35c89679-3b93-4575-a19f-c2300f30f78e-kube-api-access-n5bfl\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.244246 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.244200    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c89679-3b93-4575-a19f-c2300f30f78e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.244572 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.244554    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c89679-3b93-4575-a19f-c2300f30f78e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.244747 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.244730    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c89679-3b93-4575-a19f-c2300f30f78e-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.246460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.246439    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c89679-3b93-4575-a19f-c2300f30f78e-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.253117 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.253077    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bfl\" (UniqueName: \"kubernetes.io/projected/35c89679-3b93-4575-a19f-c2300f30f78e-kube-api-access-n5bfl\") pod \"isvc-tensorflow-predictor-6756f669d7-pw99w\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.373017 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.372995    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:12.500777 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.500714    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"]
May 20 16:21:12.503501 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:21:12.503459    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c89679_3b93_4575_a19f_c2300f30f78e.slice/crio-8f137b7c74869c1b1dda5837e1287fc72fad0c444b3b00c5012c5074fc816cd0 WatchSource:0}: Error finding container 8f137b7c74869c1b1dda5837e1287fc72fad0c444b3b00c5012c5074fc816cd0: Status 404 returned error can't find the container with id 8f137b7c74869c1b1dda5837e1287fc72fad0c444b3b00c5012c5074fc816cd0
May 20 16:21:12.505305 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:12.505288    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:21:13.135575 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:13.135537    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" event={"ID":"35c89679-3b93-4575-a19f-c2300f30f78e","Type":"ContainerStarted","Data":"ce002cce87cf0f43fca72d24ffc095cc09e825216750cc404aa146be369951fb"}
May 20 16:21:13.135575 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:13.135577    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" event={"ID":"35c89679-3b93-4575-a19f-c2300f30f78e","Type":"ContainerStarted","Data":"8f137b7c74869c1b1dda5837e1287fc72fad0c444b3b00c5012c5074fc816cd0"}
May 20 16:21:13.934381 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:13.934347    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
May 20 16:21:13.938730 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:13.938708    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
May 20 16:21:15.917438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.917400    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:21:15.968711 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.968639    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54ba2153-f238-430f-bd15-ac82c13570cc-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"54ba2153-f238-430f-bd15-ac82c13570cc\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") "
May 20 16:21:15.968711 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.968697    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54ba2153-f238-430f-bd15-ac82c13570cc-kserve-provision-location\") pod \"54ba2153-f238-430f-bd15-ac82c13570cc\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") "
May 20 16:21:15.968887 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.968744    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46sjl\" (UniqueName: \"kubernetes.io/projected/54ba2153-f238-430f-bd15-ac82c13570cc-kube-api-access-46sjl\") pod \"54ba2153-f238-430f-bd15-ac82c13570cc\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") "
May 20 16:21:15.968887 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.968764    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls\") pod \"54ba2153-f238-430f-bd15-ac82c13570cc\" (UID: \"54ba2153-f238-430f-bd15-ac82c13570cc\") "
May 20 16:21:15.969037 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.969013    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ba2153-f238-430f-bd15-ac82c13570cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54ba2153-f238-430f-bd15-ac82c13570cc" (UID: "54ba2153-f238-430f-bd15-ac82c13570cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:21:15.969085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.969035    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ba2153-f238-430f-bd15-ac82c13570cc-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "54ba2153-f238-430f-bd15-ac82c13570cc" (UID: "54ba2153-f238-430f-bd15-ac82c13570cc"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:21:15.970769 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.970746    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54ba2153-f238-430f-bd15-ac82c13570cc" (UID: "54ba2153-f238-430f-bd15-ac82c13570cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:21:15.970927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:15.970909    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ba2153-f238-430f-bd15-ac82c13570cc-kube-api-access-46sjl" (OuterVolumeSpecName: "kube-api-access-46sjl") pod "54ba2153-f238-430f-bd15-ac82c13570cc" (UID: "54ba2153-f238-430f-bd15-ac82c13570cc"). InnerVolumeSpecName "kube-api-access-46sjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:21:16.070169 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.070136    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54ba2153-f238-430f-bd15-ac82c13570cc-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:21:16.070169 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.070169    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54ba2153-f238-430f-bd15-ac82c13570cc-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:21:16.070353 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.070179    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46sjl\" (UniqueName: \"kubernetes.io/projected/54ba2153-f238-430f-bd15-ac82c13570cc-kube-api-access-46sjl\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:21:16.070353 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.070189    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54ba2153-f238-430f-bd15-ac82c13570cc-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:21:16.145709 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.145675    2556 generic.go:358] "Generic (PLEG): container finished" podID="54ba2153-f238-430f-bd15-ac82c13570cc" containerID="313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe" exitCode=0
May 20 16:21:16.145865 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.145738    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" event={"ID":"54ba2153-f238-430f-bd15-ac82c13570cc","Type":"ContainerDied","Data":"313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe"}
May 20 16:21:16.145865 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.145768    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"
May 20 16:21:16.145865 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.145786    2556 scope.go:117] "RemoveContainer" containerID="03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18"
May 20 16:21:16.146022 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.145774    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn" event={"ID":"54ba2153-f238-430f-bd15-ac82c13570cc","Type":"ContainerDied","Data":"d49c81ceb85d2f8e8a76dcaba297ed121e1f54023d41a6f6ee8735cc69a307b1"}
May 20 16:21:16.153364 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.153284    2556 scope.go:117] "RemoveContainer" containerID="313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe"
May 20 16:21:16.160131 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.160116    2556 scope.go:117] "RemoveContainer" containerID="2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1"
May 20 16:21:16.166573 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.166556    2556 scope.go:117] "RemoveContainer" containerID="03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18"
May 20 16:21:16.166849 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:21:16.166830    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18\": container with ID starting with 03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18 not found: ID does not exist" containerID="03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18"
May 20 16:21:16.166924 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.166855    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18"} err="failed to get container status \"03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18\": rpc error: code = NotFound desc = could not find container \"03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18\": container with ID starting with 03bbb526e7042f9fa4dd72272df23d832247a68f14aa944aafba5b96d3e34a18 not found: ID does not exist"
May 20 16:21:16.166924 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.166870    2556 scope.go:117] "RemoveContainer" containerID="313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe"
May 20 16:21:16.167110 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:21:16.167078    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe\": container with ID starting with 313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe not found: ID does not exist" containerID="313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe"
May 20 16:21:16.167168 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.167118    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe"} err="failed to get container status \"313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe\": rpc error: code = NotFound desc = could not find container \"313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe\": container with ID starting with 313324f36505d9299c33b2494a6c5e8072d5a289dfbe0d41964acca3e0e529fe not found: ID does not exist"
May 20 16:21:16.167168 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.167132    2556 scope.go:117] "RemoveContainer" containerID="2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1"
May 20 16:21:16.167324 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.167306    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"]
May 20 16:21:16.167380 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:21:16.167321    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1\": container with ID starting with 2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1 not found: ID does not exist" containerID="2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1"
May 20 16:21:16.167380 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.167340    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1"} err="failed to get container status \"2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1\": rpc error: code = NotFound desc = could not find container \"2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1\": container with ID starting with 2660960ee76f72856d0b5d2dcb2c64558e3468b8d3893a8600292747ce0cfca1 not found: ID does not exist"
May 20 16:21:16.170934 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:16.170914    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-59c5d55f68-c96wn"]
May 20 16:21:17.150628 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:17.150559    2556 generic.go:358] "Generic (PLEG): container finished" podID="35c89679-3b93-4575-a19f-c2300f30f78e" containerID="ce002cce87cf0f43fca72d24ffc095cc09e825216750cc404aa146be369951fb" exitCode=0
May 20 16:21:17.150628 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:17.150600    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" event={"ID":"35c89679-3b93-4575-a19f-c2300f30f78e","Type":"ContainerDied","Data":"ce002cce87cf0f43fca72d24ffc095cc09e825216750cc404aa146be369951fb"}
May 20 16:21:17.896402 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:17.896331    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" path="/var/lib/kubelet/pods/54ba2153-f238-430f-bd15-ac82c13570cc/volumes"
May 20 16:21:21.163954 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:21.163931    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" event={"ID":"35c89679-3b93-4575-a19f-c2300f30f78e","Type":"ContainerStarted","Data":"6455b7d5f6767a194ffad5a76c3b083f55d9494116f3a11d7431a5b3c79a6e87"}
May 20 16:21:22.168454 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:22.168418    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" event={"ID":"35c89679-3b93-4575-a19f-c2300f30f78e","Type":"ContainerStarted","Data":"c32115de34f515e72ad55647a40c5c6c08e9116f203bb03486a67592ef3d87b0"}
May 20 16:21:22.168850 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:22.168677    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:22.188801 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:22.188751    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podStartSLOduration=6.270768697 podStartE2EDuration="10.188740261s" podCreationTimestamp="2026-05-20 16:21:12 +0000 UTC" firstStartedPulling="2026-05-20 16:21:17.151658143 +0000 UTC m=+2919.842914048" lastFinishedPulling="2026-05-20 16:21:21.069629691 +0000 UTC m=+2923.760885612" observedRunningTime="2026-05-20 16:21:22.188439106 +0000 UTC m=+2924.879695030" watchObservedRunningTime="2026-05-20 16:21:22.188740261 +0000 UTC m=+2924.879996190"
May 20 16:21:23.171389 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:23.171357    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:23.172592 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:23.172567    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
May 20 16:21:24.173378 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:24.173335    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
May 20 16:21:29.178116 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:29.178074    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:29.178631 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:29.178605    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
May 20 16:21:39.179148 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:39.179090    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:21:51.743197 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.743159    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"]
May 20 16:21:51.743643 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.743531    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kserve-container" containerID="cri-o://6455b7d5f6767a194ffad5a76c3b083f55d9494116f3a11d7431a5b3c79a6e87" gracePeriod=30
May 20 16:21:51.743643 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.743561    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy" containerID="cri-o://c32115de34f515e72ad55647a40c5c6c08e9116f203bb03486a67592ef3d87b0" gracePeriod=30
May 20 16:21:51.836593 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836563    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"]
May 20 16:21:51.836834 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836821    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container"
May 20 16:21:51.836883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836836    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container"
May 20 16:21:51.836883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836848    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="storage-initializer"
May 20 16:21:51.836883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836854    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="storage-initializer"
May 20 16:21:51.836883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836862    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kube-rbac-proxy"
May 20 16:21:51.836883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836868    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kube-rbac-proxy"
May 20 16:21:51.837039 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836919    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kserve-container"
May 20 16:21:51.837039 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.836928    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="54ba2153-f238-430f-bd15-ac82c13570cc" containerName="kube-rbac-proxy"
May 20 16:21:51.838933 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.838918    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:51.841466 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.841446    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
May 20 16:21:51.841561 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.841449    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
May 20 16:21:51.850207 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.850179    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"]
May 20 16:21:51.917322 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.917304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/341af0f9-aa28-4b66-80e9-0ab480c3bf52-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:51.917417 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.917341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhqs\" (UniqueName: \"kubernetes.io/projected/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kube-api-access-kxhqs\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:51.917417 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.917362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:51.917506 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:51.917462    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/341af0f9-aa28-4b66-80e9-0ab480c3bf52-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.018667 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.018612    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhqs\" (UniqueName: \"kubernetes.io/projected/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kube-api-access-kxhqs\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.018667 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.018644    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.018776 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.018701    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/341af0f9-aa28-4b66-80e9-0ab480c3bf52-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.018828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.018803    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/341af0f9-aa28-4b66-80e9-0ab480c3bf52-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.019197 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.019170    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.019489 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.019465    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/341af0f9-aa28-4b66-80e9-0ab480c3bf52-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.021474 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.021452    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/341af0f9-aa28-4b66-80e9-0ab480c3bf52-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.027471 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.027448    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhqs\" (UniqueName: \"kubernetes.io/projected/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kube-api-access-kxhqs\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.149578 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.149548    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:52.257509 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.257454    2556 generic.go:358] "Generic (PLEG): container finished" podID="35c89679-3b93-4575-a19f-c2300f30f78e" containerID="c32115de34f515e72ad55647a40c5c6c08e9116f203bb03486a67592ef3d87b0" exitCode=2
May 20 16:21:52.257656 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.257522    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" event={"ID":"35c89679-3b93-4575-a19f-c2300f30f78e","Type":"ContainerDied","Data":"c32115de34f515e72ad55647a40c5c6c08e9116f203bb03486a67592ef3d87b0"}
May 20 16:21:52.278724 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:52.278661    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"]
May 20 16:21:52.281672 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:21:52.281643    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341af0f9_aa28_4b66_80e9_0ab480c3bf52.slice/crio-addb5e9789d68baddc7dbe22a769ef268b4e5f0449386a99dd9034ac5ccdca66 WatchSource:0}: Error finding container addb5e9789d68baddc7dbe22a769ef268b4e5f0449386a99dd9034ac5ccdca66: Status 404 returned error can't find the container with id addb5e9789d68baddc7dbe22a769ef268b4e5f0449386a99dd9034ac5ccdca66
May 20 16:21:53.261531 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:53.261491    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" event={"ID":"341af0f9-aa28-4b66-80e9-0ab480c3bf52","Type":"ContainerStarted","Data":"79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55"}
May 20 16:21:53.261531 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:53.261532    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" event={"ID":"341af0f9-aa28-4b66-80e9-0ab480c3bf52","Type":"ContainerStarted","Data":"addb5e9789d68baddc7dbe22a769ef268b4e5f0449386a99dd9034ac5ccdca66"}
May 20 16:21:54.173620 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:54.173579    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
May 20 16:21:57.275196 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:57.275159    2556 generic.go:358] "Generic (PLEG): container finished" podID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerID="79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55" exitCode=0
May 20 16:21:57.275599 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:57.275235    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" event={"ID":"341af0f9-aa28-4b66-80e9-0ab480c3bf52","Type":"ContainerDied","Data":"79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55"}
May 20 16:21:58.280012 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:58.279974    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" event={"ID":"341af0f9-aa28-4b66-80e9-0ab480c3bf52","Type":"ContainerStarted","Data":"97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af"}
May 20 16:21:58.280012 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:58.280018    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" event={"ID":"341af0f9-aa28-4b66-80e9-0ab480c3bf52","Type":"ContainerStarted","Data":"84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084"}
May 20 16:21:58.280437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:58.280246    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:58.299087 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:58.299041    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podStartSLOduration=7.299025621 podStartE2EDuration="7.299025621s" podCreationTimestamp="2026-05-20 16:21:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:21:58.297965617 +0000 UTC m=+2960.989221547" watchObservedRunningTime="2026-05-20 16:21:58.299025621 +0000 UTC m=+2960.990281549"
May 20 16:21:59.173973 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:59.173932    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
May 20 16:21:59.282979 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:59.282939    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:21:59.283964 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:21:59.283936    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
May 20 16:22:00.285870 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:00.285828    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
May 20 16:22:04.173912 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:04.173870    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
May 20 16:22:04.174396 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:04.173990    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:22:05.291438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:05.291407    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:22:05.292020 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:05.291991    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
May 20 16:22:09.174481 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:09.174439    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
May 20 16:22:14.174023 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:14.173978    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
May 20 16:22:15.292679 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:15.292652    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:22:19.173858 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:19.173817    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
May 20 16:22:22.341406 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.341377    2556 generic.go:358] "Generic (PLEG): container finished" podID="35c89679-3b93-4575-a19f-c2300f30f78e" containerID="6455b7d5f6767a194ffad5a76c3b083f55d9494116f3a11d7431a5b3c79a6e87" exitCode=137
May 20 16:22:22.341766 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.341444    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" event={"ID":"35c89679-3b93-4575-a19f-c2300f30f78e","Type":"ContainerDied","Data":"6455b7d5f6767a194ffad5a76c3b083f55d9494116f3a11d7431a5b3c79a6e87"}
May 20 16:22:22.380177 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.380155    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:22:22.542338 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.542274    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c89679-3b93-4575-a19f-c2300f30f78e-proxy-tls\") pod \"35c89679-3b93-4575-a19f-c2300f30f78e\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") "
May 20 16:22:22.542338 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.542319    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c89679-3b93-4575-a19f-c2300f30f78e-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"35c89679-3b93-4575-a19f-c2300f30f78e\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") "
May 20 16:22:22.542502 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.542342    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c89679-3b93-4575-a19f-c2300f30f78e-kserve-provision-location\") pod \"35c89679-3b93-4575-a19f-c2300f30f78e\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") "
May 20 16:22:22.542502 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.542389    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5bfl\" (UniqueName: \"kubernetes.io/projected/35c89679-3b93-4575-a19f-c2300f30f78e-kube-api-access-n5bfl\") pod \"35c89679-3b93-4575-a19f-c2300f30f78e\" (UID: \"35c89679-3b93-4575-a19f-c2300f30f78e\") "
May 20 16:22:22.542685 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.542658    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c89679-3b93-4575-a19f-c2300f30f78e-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "35c89679-3b93-4575-a19f-c2300f30f78e" (UID: "35c89679-3b93-4575-a19f-c2300f30f78e"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:22:22.544496 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.544462    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c89679-3b93-4575-a19f-c2300f30f78e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35c89679-3b93-4575-a19f-c2300f30f78e" (UID: "35c89679-3b93-4575-a19f-c2300f30f78e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:22:22.544587 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.544529    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c89679-3b93-4575-a19f-c2300f30f78e-kube-api-access-n5bfl" (OuterVolumeSpecName: "kube-api-access-n5bfl") pod "35c89679-3b93-4575-a19f-c2300f30f78e" (UID: "35c89679-3b93-4575-a19f-c2300f30f78e"). InnerVolumeSpecName "kube-api-access-n5bfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:22:22.553655 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.553633    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c89679-3b93-4575-a19f-c2300f30f78e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35c89679-3b93-4575-a19f-c2300f30f78e" (UID: "35c89679-3b93-4575-a19f-c2300f30f78e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:22:22.643179 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.643157    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5bfl\" (UniqueName: \"kubernetes.io/projected/35c89679-3b93-4575-a19f-c2300f30f78e-kube-api-access-n5bfl\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:22:22.643179 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.643176    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c89679-3b93-4575-a19f-c2300f30f78e-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:22:22.643291 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.643186    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c89679-3b93-4575-a19f-c2300f30f78e-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:22:22.643291 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:22.643196    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c89679-3b93-4575-a19f-c2300f30f78e-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:22:23.346519 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:23.346480    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w" event={"ID":"35c89679-3b93-4575-a19f-c2300f30f78e","Type":"ContainerDied","Data":"8f137b7c74869c1b1dda5837e1287fc72fad0c444b3b00c5012c5074fc816cd0"}
May 20 16:22:23.346910 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:23.346538    2556 scope.go:117] "RemoveContainer" containerID="c32115de34f515e72ad55647a40c5c6c08e9116f203bb03486a67592ef3d87b0"
May 20 16:22:23.346910 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:23.346574    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"
May 20 16:22:23.354724 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:23.354706    2556 scope.go:117] "RemoveContainer" containerID="6455b7d5f6767a194ffad5a76c3b083f55d9494116f3a11d7431a5b3c79a6e87"
May 20 16:22:23.361229 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:23.361210    2556 scope.go:117] "RemoveContainer" containerID="ce002cce87cf0f43fca72d24ffc095cc09e825216750cc404aa146be369951fb"
May 20 16:22:23.367713 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:23.367619    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"]
May 20 16:22:23.371663 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:23.371644    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-pw99w"]
May 20 16:22:23.890849 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:23.890815    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" path="/var/lib/kubelet/pods/35c89679-3b93-4575-a19f-c2300f30f78e/volumes"
May 20 16:22:31.677229 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.677198    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"]
May 20 16:22:31.677838 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.677522    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kserve-container" containerID="cri-o://84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084" gracePeriod=30
May 20 16:22:31.677838 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.677595    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy" containerID="cri-o://97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af" gracePeriod=30
May 20 16:22:31.733382 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733358    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"]
May 20 16:22:31.733616 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733604    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy"
May 20 16:22:31.733661 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733618    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy"
May 20 16:22:31.733661 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733631    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="storage-initializer"
May 20 16:22:31.733661 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733637    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="storage-initializer"
May 20 16:22:31.733661 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733642    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kserve-container"
May 20 16:22:31.733661 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733648    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kserve-container"
May 20 16:22:31.733818 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733692    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kserve-container"
May 20 16:22:31.733818 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.733701    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="35c89679-3b93-4575-a19f-c2300f30f78e" containerName="kube-rbac-proxy"
May 20 16:22:31.738063 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.738049    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:31.740918 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.740898    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
May 20 16:22:31.740998 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.740965    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
May 20 16:22:31.746266 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.746249    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"]
May 20 16:22:31.911433 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.911404    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:31.911601 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.911457    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:31.911601 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.911511    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lfc\" (UniqueName: \"kubernetes.io/projected/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kube-api-access-t8lfc\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:31.911601 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:31.911531    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.012781 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.012745    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.012956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.012800    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lfc\" (UniqueName: \"kubernetes.io/projected/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kube-api-access-t8lfc\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.012956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.012827    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.012956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.012873    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.013302 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.013205    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.013565 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.013542    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.015251 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.015234    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.021825 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.021807    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lfc\" (UniqueName: \"kubernetes.io/projected/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kube-api-access-t8lfc\") pod \"isvc-triton-predictor-84bb65d94b-qmd9x\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.048679 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.048657    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:22:32.173040 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.173013    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"]
May 20 16:22:32.175572 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:22:32.175552    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f66c32_3a63_4f0f_a71e_4986ba2a6fff.slice/crio-3ea1c4daf7c6078e49cbdbfd8cc1126e7517fb82ad69447e84eae05cc3ca89be WatchSource:0}: Error finding container 3ea1c4daf7c6078e49cbdbfd8cc1126e7517fb82ad69447e84eae05cc3ca89be: Status 404 returned error can't find the container with id 3ea1c4daf7c6078e49cbdbfd8cc1126e7517fb82ad69447e84eae05cc3ca89be
May 20 16:22:32.375137 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.375019    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" event={"ID":"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff","Type":"ContainerStarted","Data":"020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c"}
May 20 16:22:32.375137 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.375060    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" event={"ID":"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff","Type":"ContainerStarted","Data":"3ea1c4daf7c6078e49cbdbfd8cc1126e7517fb82ad69447e84eae05cc3ca89be"}
May 20 16:22:32.376941 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.376914    2556 generic.go:358] "Generic (PLEG): container finished" podID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerID="97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af" exitCode=2
May 20 16:22:32.377077 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:32.376950    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" event={"ID":"341af0f9-aa28-4b66-80e9-0ab480c3bf52","Type":"ContainerDied","Data":"97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af"}
May 20 16:22:35.286078 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:35.286040    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
May 20 16:22:36.389247 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:36.389215    2556 generic.go:358] "Generic (PLEG): container finished" podID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerID="020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c" exitCode=0
May 20 16:22:36.389643 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:36.389268    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" event={"ID":"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff","Type":"ContainerDied","Data":"020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c"}
May 20 16:22:40.286291 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:40.286248    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
May 20 16:22:45.287000 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:45.286292    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
May 20 16:22:45.287000 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:45.286450    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:22:50.286546 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:50.286488    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
May 20 16:22:55.286376 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:22:55.286324    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
May 20 16:23:00.286414 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:00.286360    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
May 20 16:23:02.370240 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.370213    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:23:02.475883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.475793    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/341af0f9-aa28-4b66-80e9-0ab480c3bf52-proxy-tls\") pod \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") "
May 20 16:23:02.475883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.475843    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/341af0f9-aa28-4b66-80e9-0ab480c3bf52-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") "
May 20 16:23:02.475883 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.475882    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxhqs\" (UniqueName: \"kubernetes.io/projected/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kube-api-access-kxhqs\") pod \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") "
May 20 16:23:02.476389 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.475950    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kserve-provision-location\") pod \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\" (UID: \"341af0f9-aa28-4b66-80e9-0ab480c3bf52\") "
May 20 16:23:02.476453 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.476375    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341af0f9-aa28-4b66-80e9-0ab480c3bf52-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "341af0f9-aa28-4b66-80e9-0ab480c3bf52" (UID: "341af0f9-aa28-4b66-80e9-0ab480c3bf52"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:23:02.478958 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.478923    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kube-api-access-kxhqs" (OuterVolumeSpecName: "kube-api-access-kxhqs") pod "341af0f9-aa28-4b66-80e9-0ab480c3bf52" (UID: "341af0f9-aa28-4b66-80e9-0ab480c3bf52"). InnerVolumeSpecName "kube-api-access-kxhqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:23:02.479240 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.479197    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341af0f9-aa28-4b66-80e9-0ab480c3bf52-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "341af0f9-aa28-4b66-80e9-0ab480c3bf52" (UID: "341af0f9-aa28-4b66-80e9-0ab480c3bf52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:23:02.480650 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.480624    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "341af0f9-aa28-4b66-80e9-0ab480c3bf52" (UID: "341af0f9-aa28-4b66-80e9-0ab480c3bf52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:23:02.491217 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.491188    2556 generic.go:358] "Generic (PLEG): container finished" podID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerID="84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084" exitCode=137
May 20 16:23:02.491322 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.491231    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" event={"ID":"341af0f9-aa28-4b66-80e9-0ab480c3bf52","Type":"ContainerDied","Data":"84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084"}
May 20 16:23:02.491322 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.491259    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd" event={"ID":"341af0f9-aa28-4b66-80e9-0ab480c3bf52","Type":"ContainerDied","Data":"addb5e9789d68baddc7dbe22a769ef268b4e5f0449386a99dd9034ac5ccdca66"}
May 20 16:23:02.491322 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.491277    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"
May 20 16:23:02.491482 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.491279    2556 scope.go:117] "RemoveContainer" containerID="97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af"
May 20 16:23:02.501200 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.501173    2556 scope.go:117] "RemoveContainer" containerID="84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084"
May 20 16:23:02.509886 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.509816    2556 scope.go:117] "RemoveContainer" containerID="79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55"
May 20 16:23:02.518421 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.518397    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"]
May 20 16:23:02.519771 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.519704    2556 scope.go:117] "RemoveContainer" containerID="97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af"
May 20 16:23:02.519998 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:23:02.519954    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af\": container with ID starting with 97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af not found: ID does not exist" containerID="97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af"
May 20 16:23:02.520085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.519988    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af"} err="failed to get container status \"97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af\": rpc error: code = NotFound desc = could not find container \"97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af\": container with ID starting with 97f51dd019485a023f6c54165ff3faef89ce276874411b93ff5b02036a20d1af not found: ID does not exist"
May 20 16:23:02.520085 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.520015    2556 scope.go:117] "RemoveContainer" containerID="84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084"
May 20 16:23:02.520330 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:23:02.520307    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084\": container with ID starting with 84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084 not found: ID does not exist" containerID="84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084"
May 20 16:23:02.520454 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.520338    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084"} err="failed to get container status \"84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084\": rpc error: code = NotFound desc = could not find container \"84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084\": container with ID starting with 84326a59de309bde915564836e776cb3141fd2053b0d590c1f58bc879350a084 not found: ID does not exist"
May 20 16:23:02.520454 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.520359    2556 scope.go:117] "RemoveContainer" containerID="79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55"
May 20 16:23:02.520652 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:23:02.520627    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55\": container with ID starting with 79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55 not found: ID does not exist" containerID="79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55"
May 20 16:23:02.520746 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.520661    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55"} err="failed to get container status \"79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55\": rpc error: code = NotFound desc = could not find container \"79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55\": container with ID starting with 79f2495812e2fc2f5a9da328e8ccea65d5b214fe1c06b3f516a2401093d94d55 not found: ID does not exist"
May 20 16:23:02.521118 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.521048    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-rf5cd"]
May 20 16:23:02.577387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.577345    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:23:02.577387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.577370    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/341af0f9-aa28-4b66-80e9-0ab480c3bf52-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:23:02.577387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.577387    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/341af0f9-aa28-4b66-80e9-0ab480c3bf52-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:23:02.577604 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:02.577400    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhqs\" (UniqueName: \"kubernetes.io/projected/341af0f9-aa28-4b66-80e9-0ab480c3bf52-kube-api-access-kxhqs\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:23:03.892468 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:23:03.892412    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" path="/var/lib/kubelet/pods/341af0f9-aa28-4b66-80e9-0ab480c3bf52/volumes"
May 20 16:24:30.752265 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:30.752238    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" event={"ID":"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff","Type":"ContainerStarted","Data":"930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b"}
May 20 16:24:31.756919 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:31.756879    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" event={"ID":"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff","Type":"ContainerStarted","Data":"460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6"}
May 20 16:24:31.757327 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:31.757027    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:24:31.779873 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:31.779830    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" podStartSLOduration=6.589334787 podStartE2EDuration="2m0.779820444s" podCreationTimestamp="2026-05-20 16:22:31 +0000 UTC" firstStartedPulling="2026-05-20 16:22:36.390256373 +0000 UTC m=+2999.081512279" lastFinishedPulling="2026-05-20 16:24:30.580742029 +0000 UTC m=+3113.271997936" observedRunningTime="2026-05-20 16:24:31.778553247 +0000 UTC m=+3114.469809176" watchObservedRunningTime="2026-05-20 16:24:31.779820444 +0000 UTC m=+3114.471076372"
May 20 16:24:32.759853 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:32.759820    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:24:32.760889 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:32.760865    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
May 20 16:24:33.762498 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:33.762462    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
May 20 16:24:38.773291 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:38.773247    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:24:38.774281 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:38.774257    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:24:43.173376 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.173346    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"]
May 20 16:24:43.173784 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.173627    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kserve-container" containerID="cri-o://930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b" gracePeriod=30
May 20 16:24:43.173784 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.173675    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kube-rbac-proxy" containerID="cri-o://460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6" gracePeriod=30
May 20 16:24:43.283574 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283550    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"]
May 20 16:24:43.283808 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283797    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy"
May 20 16:24:43.283853 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283810    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy"
May 20 16:24:43.283853 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283821    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="storage-initializer"
May 20 16:24:43.283853 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283827    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="storage-initializer"
May 20 16:24:43.283853 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283834    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kserve-container"
May 20 16:24:43.283853 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283840    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kserve-container"
May 20 16:24:43.284036 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283891    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kube-rbac-proxy"
May 20 16:24:43.284036 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.283901    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="341af0f9-aa28-4b66-80e9-0ab480c3bf52" containerName="kserve-container"
May 20 16:24:43.299123 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.298026    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.302061 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.302039    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
May 20 16:24:43.302206 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.302116    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
May 20 16:24:43.302285 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.302258    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"]
May 20 16:24:43.383277 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.383246    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.383459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.383299    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.383459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.383360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527rx\" (UniqueName: \"kubernetes.io/projected/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kube-api-access-527rx\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.383459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.383391    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.484608 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.484586    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.484755 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.484635    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.484755 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.484654    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-527rx\" (UniqueName: \"kubernetes.io/projected/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kube-api-access-527rx\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.484755 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.484670    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.485029 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.485011    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.485221 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.485202    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.486914 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.486891    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.493296 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.493275    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-527rx\" (UniqueName: \"kubernetes.io/projected/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kube-api-access-527rx\") pod \"isvc-xgboost-predictor-8689c4cfcc-6t97l\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.609237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.609209    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:24:43.731828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.731742    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"]
May 20 16:24:43.733846 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:24:43.733819    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded39edba_0dc4_4d75_8d17_725d1bb82a8a.slice/crio-bd92519e83c5642d6a2e5fd7a4b5997da507a9b2fa0cf689a7e493a8440de856 WatchSource:0}: Error finding container bd92519e83c5642d6a2e5fd7a4b5997da507a9b2fa0cf689a7e493a8440de856: Status 404 returned error can't find the container with id bd92519e83c5642d6a2e5fd7a4b5997da507a9b2fa0cf689a7e493a8440de856
May 20 16:24:43.763207 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.763180    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
May 20 16:24:43.790053 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.790030    2556 generic.go:358] "Generic (PLEG): container finished" podID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerID="460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6" exitCode=2
May 20 16:24:43.790181 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.790076    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" event={"ID":"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff","Type":"ContainerDied","Data":"460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6"}
May 20 16:24:43.791120 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:43.791078    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" event={"ID":"ed39edba-0dc4-4d75-8d17-725d1bb82a8a","Type":"ContainerStarted","Data":"bd92519e83c5642d6a2e5fd7a4b5997da507a9b2fa0cf689a7e493a8440de856"}
May 20 16:24:44.794567 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:44.794531    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" event={"ID":"ed39edba-0dc4-4d75-8d17-725d1bb82a8a","Type":"ContainerStarted","Data":"c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1"}
May 20 16:24:45.230494 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.230471    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:24:45.298292 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.298260    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kserve-provision-location\") pod \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") "
May 20 16:24:45.298457 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.298309    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8lfc\" (UniqueName: \"kubernetes.io/projected/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kube-api-access-t8lfc\") pod \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") "
May 20 16:24:45.298457 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.298338    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-isvc-triton-kube-rbac-proxy-sar-config\") pod \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") "
May 20 16:24:45.298457 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.298371    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-proxy-tls\") pod \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\" (UID: \"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff\") "
May 20 16:24:45.298719 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.298697    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" (UID: "c6f66c32-3a63-4f0f-a71e-4986ba2a6fff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:24:45.298809 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.298762    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" (UID: "c6f66c32-3a63-4f0f-a71e-4986ba2a6fff"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:24:45.300462 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.300432    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" (UID: "c6f66c32-3a63-4f0f-a71e-4986ba2a6fff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:24:45.300554 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.300525    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kube-api-access-t8lfc" (OuterVolumeSpecName: "kube-api-access-t8lfc") pod "c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" (UID: "c6f66c32-3a63-4f0f-a71e-4986ba2a6fff"). InnerVolumeSpecName "kube-api-access-t8lfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:24:45.399920 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.399833    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8lfc\" (UniqueName: \"kubernetes.io/projected/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kube-api-access-t8lfc\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:24:45.399920 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.399861    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:24:45.399920 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.399877    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:24:45.399920 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.399889    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:24:45.799157 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.799123    2556 generic.go:358] "Generic (PLEG): container finished" podID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerID="930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b" exitCode=0
May 20 16:24:45.799613 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.799191    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" event={"ID":"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff","Type":"ContainerDied","Data":"930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b"}
May 20 16:24:45.799613 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.799206    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"
May 20 16:24:45.799613 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.799243    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x" event={"ID":"c6f66c32-3a63-4f0f-a71e-4986ba2a6fff","Type":"ContainerDied","Data":"3ea1c4daf7c6078e49cbdbfd8cc1126e7517fb82ad69447e84eae05cc3ca89be"}
May 20 16:24:45.799613 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.799262    2556 scope.go:117] "RemoveContainer" containerID="460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6"
May 20 16:24:45.808129 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.808084    2556 scope.go:117] "RemoveContainer" containerID="930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b"
May 20 16:24:45.815502 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.815481    2556 scope.go:117] "RemoveContainer" containerID="020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c"
May 20 16:24:45.820820 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.820796    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"]
May 20 16:24:45.822794 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.822774    2556 scope.go:117] "RemoveContainer" containerID="460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6"
May 20 16:24:45.823146 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:24:45.823121    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6\": container with ID starting with 460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6 not found: ID does not exist" containerID="460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6"
May 20 16:24:45.823254 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.823157    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6"} err="failed to get container status \"460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6\": rpc error: code = NotFound desc = could not find container \"460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6\": container with ID starting with 460a12ae8b4eb4d4dbb7806e90ded2183b1f79c0e0beb0da99488f49c7d5e0c6 not found: ID does not exist"
May 20 16:24:45.823254 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.823180    2556 scope.go:117] "RemoveContainer" containerID="930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b"
May 20 16:24:45.823461 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:24:45.823441    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b\": container with ID starting with 930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b not found: ID does not exist" containerID="930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b"
May 20 16:24:45.823527 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.823466    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b"} err="failed to get container status \"930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b\": rpc error: code = NotFound desc = could not find container \"930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b\": container with ID starting with 930b0169556c8d7b61727d9212a1eefc36148042449d9a9650e46000d105790b not found: ID does not exist"
May 20 16:24:45.823527 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.823485    2556 scope.go:117] "RemoveContainer" containerID="020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c"
May 20 16:24:45.823808 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:24:45.823749    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c\": container with ID starting with 020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c not found: ID does not exist" containerID="020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c"
May 20 16:24:45.823808 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.823779    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c"} err="failed to get container status \"020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c\": rpc error: code = NotFound desc = could not find container \"020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c\": container with ID starting with 020a47b2be5680598c988ac4f1a8509c431aca895f5bb796254b870d4000c28c not found: ID does not exist"
May 20 16:24:45.825253 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.825232    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-qmd9x"]
May 20 16:24:45.890849 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:45.890828    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" path="/var/lib/kubelet/pods/c6f66c32-3a63-4f0f-a71e-4986ba2a6fff/volumes"
May 20 16:24:48.809431 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:48.809392    2556 generic.go:358] "Generic (PLEG): container finished" podID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerID="c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1" exitCode=0
May 20 16:24:48.809820 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:24:48.809461    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" event={"ID":"ed39edba-0dc4-4d75-8d17-725d1bb82a8a","Type":"ContainerDied","Data":"c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1"}
May 20 16:25:11.874742 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:11.874708    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" event={"ID":"ed39edba-0dc4-4d75-8d17-725d1bb82a8a","Type":"ContainerStarted","Data":"7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e"}
May 20 16:25:11.874742 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:11.874749    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" event={"ID":"ed39edba-0dc4-4d75-8d17-725d1bb82a8a","Type":"ContainerStarted","Data":"717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e"}
May 20 16:25:11.875283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:11.875043    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:25:11.875283 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:11.875185    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:25:11.876338 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:11.876309    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
May 20 16:25:11.895943 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:11.895897    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podStartSLOduration=6.495609195 podStartE2EDuration="28.895885641s" podCreationTimestamp="2026-05-20 16:24:43 +0000 UTC" firstStartedPulling="2026-05-20 16:24:48.810647194 +0000 UTC m=+3131.501903100" lastFinishedPulling="2026-05-20 16:25:11.210923637 +0000 UTC m=+3153.902179546" observedRunningTime="2026-05-20 16:25:11.894370464 +0000 UTC m=+3154.585626393" watchObservedRunningTime="2026-05-20 16:25:11.895885641 +0000 UTC m=+3154.587141568"
May 20 16:25:12.877787 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:12.877746    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
May 20 16:25:17.882022 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:17.881996    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:25:17.882503 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:17.882478    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
May 20 16:25:27.882946 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:27.882905    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
May 20 16:25:37.883004 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:37.882967    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
May 20 16:25:47.882602 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:47.882562    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
May 20 16:25:57.882547 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:25:57.882513    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
May 20 16:26:07.883978 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:07.883950    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:26:13.435632 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.435601    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"]
May 20 16:26:13.436128 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.435882    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container" containerID="cri-o://717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e" gracePeriod=30
May 20 16:26:13.436128 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.436082    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kube-rbac-proxy" containerID="cri-o://7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e" gracePeriod=30
May 20 16:26:13.529373 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529340    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"]
May 20 16:26:13.529658 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529641    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kserve-container"
May 20 16:26:13.529746 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529660    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kserve-container"
May 20 16:26:13.529746 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529687    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kube-rbac-proxy"
May 20 16:26:13.529746 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529695    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kube-rbac-proxy"
May 20 16:26:13.529746 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529712    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="storage-initializer"
May 20 16:26:13.529746 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529722    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="storage-initializer"
May 20 16:26:13.530007 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529785    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kserve-container"
May 20 16:26:13.530007 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.529799    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c6f66c32-3a63-4f0f-a71e-4986ba2a6fff" containerName="kube-rbac-proxy"
May 20 16:26:13.531799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.531782    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.533900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.533881    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 20 16:26:13.534005 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.533880    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
May 20 16:26:13.542938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.542917    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"]
May 20 16:26:13.726449 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.726380    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnm5f\" (UniqueName: \"kubernetes.io/projected/def02af4-057d-417b-9743-f659f204516c-kube-api-access-qnm5f\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.726449 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.726422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/def02af4-057d-417b-9743-f659f204516c-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.726449 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.726447    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/def02af4-057d-417b-9743-f659f204516c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.726648 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.726483    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def02af4-057d-417b-9743-f659f204516c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.827431 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.827406    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/def02af4-057d-417b-9743-f659f204516c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.827529 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.827438    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def02af4-057d-417b-9743-f659f204516c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.827529 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.827486    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qnm5f\" (UniqueName: \"kubernetes.io/projected/def02af4-057d-417b-9743-f659f204516c-kube-api-access-qnm5f\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.827529 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.827514    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/def02af4-057d-417b-9743-f659f204516c-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.827929 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.827910    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def02af4-057d-417b-9743-f659f204516c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.828206 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.828188    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/def02af4-057d-417b-9743-f659f204516c-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.829787 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.829771    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/def02af4-057d-417b-9743-f659f204516c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.835625 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.835608    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnm5f\" (UniqueName: \"kubernetes.io/projected/def02af4-057d-417b-9743-f659f204516c-kube-api-access-qnm5f\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.841460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.841441    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:13.964683 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.964477    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"]
May 20 16:26:13.966809 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:26:13.966783    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef02af4_057d_417b_9743_f659f204516c.slice/crio-1792977a17ca95809ce46c57f7492d7f9161f6ad6c7aaee321dc7718c3e6ea0c WatchSource:0}: Error finding container 1792977a17ca95809ce46c57f7492d7f9161f6ad6c7aaee321dc7718c3e6ea0c: Status 404 returned error can't find the container with id 1792977a17ca95809ce46c57f7492d7f9161f6ad6c7aaee321dc7718c3e6ea0c
May 20 16:26:13.968483 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:13.968467    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:26:14.039659 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:14.039637    2556 generic.go:358] "Generic (PLEG): container finished" podID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerID="7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e" exitCode=2
May 20 16:26:14.039751 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:14.039706    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" event={"ID":"ed39edba-0dc4-4d75-8d17-725d1bb82a8a","Type":"ContainerDied","Data":"7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e"}
May 20 16:26:14.040952 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:14.040930    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" event={"ID":"def02af4-057d-417b-9743-f659f204516c","Type":"ContainerStarted","Data":"e1462e4684011c218c79531ac006901264a48f699eb3e39d7a88807ef8a90117"}
May 20 16:26:14.041040 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:14.040962    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" event={"ID":"def02af4-057d-417b-9743-f659f204516c","Type":"ContainerStarted","Data":"1792977a17ca95809ce46c57f7492d7f9161f6ad6c7aaee321dc7718c3e6ea0c"}
May 20 16:26:16.682067 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.682047    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:26:16.747323 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.747300    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-proxy-tls\") pod \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") "
May 20 16:26:16.747460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.747344    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") "
May 20 16:26:16.747460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.747377    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kserve-provision-location\") pod \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") "
May 20 16:26:16.747460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.747405    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-527rx\" (UniqueName: \"kubernetes.io/projected/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kube-api-access-527rx\") pod \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\" (UID: \"ed39edba-0dc4-4d75-8d17-725d1bb82a8a\") "
May 20 16:26:16.747688 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.747664    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed39edba-0dc4-4d75-8d17-725d1bb82a8a" (UID: "ed39edba-0dc4-4d75-8d17-725d1bb82a8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:26:16.747728 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.747698    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "ed39edba-0dc4-4d75-8d17-725d1bb82a8a" (UID: "ed39edba-0dc4-4d75-8d17-725d1bb82a8a"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:26:16.749373 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.749353    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kube-api-access-527rx" (OuterVolumeSpecName: "kube-api-access-527rx") pod "ed39edba-0dc4-4d75-8d17-725d1bb82a8a" (UID: "ed39edba-0dc4-4d75-8d17-725d1bb82a8a"). InnerVolumeSpecName "kube-api-access-527rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:26:16.749468 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.749450    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed39edba-0dc4-4d75-8d17-725d1bb82a8a" (UID: "ed39edba-0dc4-4d75-8d17-725d1bb82a8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:26:16.847877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.847811    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:26:16.847877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.847836    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:26:16.847877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.847850    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-527rx\" (UniqueName: \"kubernetes.io/projected/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-kube-api-access-527rx\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:26:16.847877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:16.847865    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed39edba-0dc4-4d75-8d17-725d1bb82a8a-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:26:17.050690 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.050654    2556 generic.go:358] "Generic (PLEG): container finished" podID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerID="717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e" exitCode=0
May 20 16:26:17.050843 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.050708    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" event={"ID":"ed39edba-0dc4-4d75-8d17-725d1bb82a8a","Type":"ContainerDied","Data":"717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e"}
May 20 16:26:17.050843 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.050740    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l" event={"ID":"ed39edba-0dc4-4d75-8d17-725d1bb82a8a","Type":"ContainerDied","Data":"bd92519e83c5642d6a2e5fd7a4b5997da507a9b2fa0cf689a7e493a8440de856"}
May 20 16:26:17.050843 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.050746    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"
May 20 16:26:17.050843 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.050763    2556 scope.go:117] "RemoveContainer" containerID="7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e"
May 20 16:26:17.058628 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.058604    2556 scope.go:117] "RemoveContainer" containerID="717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e"
May 20 16:26:17.065197 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.065180    2556 scope.go:117] "RemoveContainer" containerID="c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1"
May 20 16:26:17.071565 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.071544    2556 scope.go:117] "RemoveContainer" containerID="7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e"
May 20 16:26:17.071926 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:26:17.071820    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e\": container with ID starting with 7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e not found: ID does not exist" containerID="7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e"
May 20 16:26:17.071926 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.071857    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e"} err="failed to get container status \"7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e\": rpc error: code = NotFound desc = could not find container \"7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e\": container with ID starting with 7877f4e0e7bf0a398d2fe1fe37079a72dc7c79f1d187e7c6e3acfdb9bcc4b64e not found: ID does not exist"
May 20 16:26:17.071926 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.071880    2556 scope.go:117] "RemoveContainer" containerID="717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e"
May 20 16:26:17.072179 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:26:17.072157    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e\": container with ID starting with 717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e not found: ID does not exist" containerID="717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e"
May 20 16:26:17.072226 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.072187    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e"} err="failed to get container status \"717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e\": rpc error: code = NotFound desc = could not find container \"717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e\": container with ID starting with 717f5fd3c0fe0ee122bb65856cecba926a4b96a4ad69337555cc4f3cb2fc220e not found: ID does not exist"
May 20 16:26:17.072226 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.072203    2556 scope.go:117] "RemoveContainer" containerID="c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1"
May 20 16:26:17.072485 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:26:17.072463    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1\": container with ID starting with c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1 not found: ID does not exist" containerID="c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1"
May 20 16:26:17.072527 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.072494    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1"} err="failed to get container status \"c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1\": rpc error: code = NotFound desc = could not find container \"c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1\": container with ID starting with c264a412c6a19d3a7add4aadb2cf70b436c8bad3a7400933673b1001a63f2bf1 not found: ID does not exist"
May 20 16:26:17.072949 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.072934    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"]
May 20 16:26:17.077573 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.077555    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-6t97l"]
May 20 16:26:17.890396 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:17.890366    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" path="/var/lib/kubelet/pods/ed39edba-0dc4-4d75-8d17-725d1bb82a8a/volumes"
May 20 16:26:18.055669 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:18.055637    2556 generic.go:358] "Generic (PLEG): container finished" podID="def02af4-057d-417b-9743-f659f204516c" containerID="e1462e4684011c218c79531ac006901264a48f699eb3e39d7a88807ef8a90117" exitCode=0
May 20 16:26:18.055825 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:18.055694    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" event={"ID":"def02af4-057d-417b-9743-f659f204516c","Type":"ContainerDied","Data":"e1462e4684011c218c79531ac006901264a48f699eb3e39d7a88807ef8a90117"}
May 20 16:26:19.061183 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:19.061148    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" event={"ID":"def02af4-057d-417b-9743-f659f204516c","Type":"ContainerStarted","Data":"e3ce929c753288de0d93508c8380716ba6020c5e9fb42d8a1ef3de3b41b7f955"}
May 20 16:26:19.061640 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:19.061189    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" event={"ID":"def02af4-057d-417b-9743-f659f204516c","Type":"ContainerStarted","Data":"10d9129097c22da86bc5c802a5b66eee940ccadea89bf718232a45b9cfb0de2d"}
May 20 16:26:19.061640 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:19.061448    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:19.081829 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:19.081781    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" podStartSLOduration=6.081770322 podStartE2EDuration="6.081770322s" podCreationTimestamp="2026-05-20 16:26:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:26:19.079753614 +0000 UTC m=+3221.771009542" watchObservedRunningTime="2026-05-20 16:26:19.081770322 +0000 UTC m=+3221.773026249"
May 20 16:26:20.064228 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:20.064201    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:26.071973 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:26.071943    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:26:56.076360 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:26:56.076327    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:27:03.593014 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.592981    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"]
May 20 16:27:03.593419 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.593295    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kserve-container" containerID="cri-o://10d9129097c22da86bc5c802a5b66eee940ccadea89bf718232a45b9cfb0de2d" gracePeriod=30
May 20 16:27:03.593419 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.593319    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kube-rbac-proxy" containerID="cri-o://e3ce929c753288de0d93508c8380716ba6020c5e9fb42d8a1ef3de3b41b7f955" gracePeriod=30
May 20 16:27:03.693722 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.693689    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"]
May 20 16:27:03.693939 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.693928    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="storage-initializer"
May 20 16:27:03.693992 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.693940    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="storage-initializer"
May 20 16:27:03.693992 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.693952    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kube-rbac-proxy"
May 20 16:27:03.693992 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.693958    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kube-rbac-proxy"
May 20 16:27:03.693992 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.693972    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container"
May 20 16:27:03.693992 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.693978    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container"
May 20 16:27:03.694173 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.694024    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kserve-container"
May 20 16:27:03.694173 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.694032    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed39edba-0dc4-4d75-8d17-725d1bb82a8a" containerName="kube-rbac-proxy"
May 20 16:27:03.696549 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.696525    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.698956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.698933    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 20 16:27:03.699073 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.699046    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
May 20 16:27:03.707089 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.707067    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"]
May 20 16:27:03.761930 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.761908    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ef788c-dd66-46c1-b959-58b7193d8bc0-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.762203 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.761947    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7kj\" (UniqueName: \"kubernetes.io/projected/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kube-api-access-5l7kj\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.762203 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.761989    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.762203 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.762020    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ef788c-dd66-46c1-b959-58b7193d8bc0-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.863357 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.863306    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ef788c-dd66-46c1-b959-58b7193d8bc0-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.863357 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.863344    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7kj\" (UniqueName: \"kubernetes.io/projected/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kube-api-access-5l7kj\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.863475 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.863375    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.863475 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.863415    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ef788c-dd66-46c1-b959-58b7193d8bc0-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.863822 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.863802    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.864030 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.864011    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ef788c-dd66-46c1-b959-58b7193d8bc0-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.865604 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.865585    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ef788c-dd66-46c1-b959-58b7193d8bc0-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:03.873673 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:03.873654    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7kj\" (UniqueName: \"kubernetes.io/projected/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kube-api-access-5l7kj\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-25wv7\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:04.007937 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:04.007915    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:04.133022 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:04.132946    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"]
May 20 16:27:04.136229 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:27:04.136201    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ef788c_dd66_46c1_b959_58b7193d8bc0.slice/crio-f1b30cc8aa5448fe89ffe8d0310fc8ae334250df09cb7bf24fa0ad05abb5d373 WatchSource:0}: Error finding container f1b30cc8aa5448fe89ffe8d0310fc8ae334250df09cb7bf24fa0ad05abb5d373: Status 404 returned error can't find the container with id f1b30cc8aa5448fe89ffe8d0310fc8ae334250df09cb7bf24fa0ad05abb5d373
May 20 16:27:04.187198 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:04.187175    2556 generic.go:358] "Generic (PLEG): container finished" podID="def02af4-057d-417b-9743-f659f204516c" containerID="e3ce929c753288de0d93508c8380716ba6020c5e9fb42d8a1ef3de3b41b7f955" exitCode=2
May 20 16:27:04.187288 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:04.187249    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" event={"ID":"def02af4-057d-417b-9743-f659f204516c","Type":"ContainerDied","Data":"e3ce929c753288de0d93508c8380716ba6020c5e9fb42d8a1ef3de3b41b7f955"}
May 20 16:27:04.188506 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:04.188486    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" event={"ID":"a0ef788c-dd66-46c1-b959-58b7193d8bc0","Type":"ContainerStarted","Data":"f1b30cc8aa5448fe89ffe8d0310fc8ae334250df09cb7bf24fa0ad05abb5d373"}
May 20 16:27:05.192583 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:05.192547    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" event={"ID":"a0ef788c-dd66-46c1-b959-58b7193d8bc0","Type":"ContainerStarted","Data":"4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d"}
May 20 16:27:06.067527 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:06.067495    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
May 20 16:27:06.072780 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:06.072758    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
May 20 16:27:08.203572 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:08.203538    2556 generic.go:358] "Generic (PLEG): container finished" podID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerID="4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d" exitCode=0
May 20 16:27:08.203967 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:08.203616    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" event={"ID":"a0ef788c-dd66-46c1-b959-58b7193d8bc0","Type":"ContainerDied","Data":"4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d"}
May 20 16:27:09.208168 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.208131    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" event={"ID":"a0ef788c-dd66-46c1-b959-58b7193d8bc0","Type":"ContainerStarted","Data":"0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71"}
May 20 16:27:09.208168 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.208172    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" event={"ID":"a0ef788c-dd66-46c1-b959-58b7193d8bc0","Type":"ContainerStarted","Data":"19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed"}
May 20 16:27:09.208627 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.208393    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:09.210362 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.210340    2556 generic.go:358] "Generic (PLEG): container finished" podID="def02af4-057d-417b-9743-f659f204516c" containerID="10d9129097c22da86bc5c802a5b66eee940ccadea89bf718232a45b9cfb0de2d" exitCode=0
May 20 16:27:09.210469 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.210390    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" event={"ID":"def02af4-057d-417b-9743-f659f204516c","Type":"ContainerDied","Data":"10d9129097c22da86bc5c802a5b66eee940ccadea89bf718232a45b9cfb0de2d"}
May 20 16:27:09.228376 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.228337    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" podStartSLOduration=6.228326061 podStartE2EDuration="6.228326061s" podCreationTimestamp="2026-05-20 16:27:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:27:09.227129522 +0000 UTC m=+3271.918385450" watchObservedRunningTime="2026-05-20 16:27:09.228326061 +0000 UTC m=+3271.919581989"
May 20 16:27:09.630815 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.630791    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:27:09.710138 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.710090    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnm5f\" (UniqueName: \"kubernetes.io/projected/def02af4-057d-417b-9743-f659f204516c-kube-api-access-qnm5f\") pod \"def02af4-057d-417b-9743-f659f204516c\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") "
May 20 16:27:09.710138 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.710141    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/def02af4-057d-417b-9743-f659f204516c-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"def02af4-057d-417b-9743-f659f204516c\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") "
May 20 16:27:09.710329 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.710164    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def02af4-057d-417b-9743-f659f204516c-kserve-provision-location\") pod \"def02af4-057d-417b-9743-f659f204516c\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") "
May 20 16:27:09.710329 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.710282    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/def02af4-057d-417b-9743-f659f204516c-proxy-tls\") pod \"def02af4-057d-417b-9743-f659f204516c\" (UID: \"def02af4-057d-417b-9743-f659f204516c\") "
May 20 16:27:09.710480 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.710458    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def02af4-057d-417b-9743-f659f204516c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "def02af4-057d-417b-9743-f659f204516c" (UID: "def02af4-057d-417b-9743-f659f204516c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:27:09.710480 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.710469    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def02af4-057d-417b-9743-f659f204516c-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "def02af4-057d-417b-9743-f659f204516c" (UID: "def02af4-057d-417b-9743-f659f204516c"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:27:09.710599 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.710563    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/def02af4-057d-417b-9743-f659f204516c-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:27:09.710599 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.710582    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/def02af4-057d-417b-9743-f659f204516c-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:27:09.712080 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.712055    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def02af4-057d-417b-9743-f659f204516c-kube-api-access-qnm5f" (OuterVolumeSpecName: "kube-api-access-qnm5f") pod "def02af4-057d-417b-9743-f659f204516c" (UID: "def02af4-057d-417b-9743-f659f204516c"). InnerVolumeSpecName "kube-api-access-qnm5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:27:09.712281 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.712262    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def02af4-057d-417b-9743-f659f204516c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "def02af4-057d-417b-9743-f659f204516c" (UID: "def02af4-057d-417b-9743-f659f204516c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:27:09.811421 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.811350    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/def02af4-057d-417b-9743-f659f204516c-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:27:09.811421 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:09.811371    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnm5f\" (UniqueName: \"kubernetes.io/projected/def02af4-057d-417b-9743-f659f204516c-kube-api-access-qnm5f\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:27:10.214632 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:10.214571    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"
May 20 16:27:10.214632 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:10.214576    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw" event={"ID":"def02af4-057d-417b-9743-f659f204516c","Type":"ContainerDied","Data":"1792977a17ca95809ce46c57f7492d7f9161f6ad6c7aaee321dc7718c3e6ea0c"}
May 20 16:27:10.214632 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:10.214625    2556 scope.go:117] "RemoveContainer" containerID="e3ce929c753288de0d93508c8380716ba6020c5e9fb42d8a1ef3de3b41b7f955"
May 20 16:27:10.215149 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:10.215032    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:10.222192 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:10.222173    2556 scope.go:117] "RemoveContainer" containerID="10d9129097c22da86bc5c802a5b66eee940ccadea89bf718232a45b9cfb0de2d"
May 20 16:27:10.229035 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:10.229016    2556 scope.go:117] "RemoveContainer" containerID="e1462e4684011c218c79531ac006901264a48f699eb3e39d7a88807ef8a90117"
May 20 16:27:10.240512 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:10.240483    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"]
May 20 16:27:10.244948 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:10.244928    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-tzwkw"]
May 20 16:27:11.890382 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:11.890335    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def02af4-057d-417b-9743-f659f204516c" path="/var/lib/kubelet/pods/def02af4-057d-417b-9743-f659f204516c/volumes"
May 20 16:27:16.223311 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:16.223280    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:46.227849 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:46.227818    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:53.822068 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.822028    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"]
May 20 16:27:53.822514 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.822460    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kserve-container" containerID="cri-o://19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed" gracePeriod=30
May 20 16:27:53.822514 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.822483    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kube-rbac-proxy" containerID="cri-o://0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71" gracePeriod=30
May 20 16:27:53.890490 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890468    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"]
May 20 16:27:53.890705 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890694    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="storage-initializer"
May 20 16:27:53.890750 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890707    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="storage-initializer"
May 20 16:27:53.890750 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890724    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kube-rbac-proxy"
May 20 16:27:53.890750 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890729    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kube-rbac-proxy"
May 20 16:27:53.890750 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890738    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kserve-container"
May 20 16:27:53.890750 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890743    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kserve-container"
May 20 16:27:53.890904 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890795    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kserve-container"
May 20 16:27:53.890904 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.890802    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="def02af4-057d-417b-9743-f659f204516c" containerName="kube-rbac-proxy"
May 20 16:27:53.892565 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.892550    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:53.894941 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.894926    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
May 20 16:27:53.895019 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.894975    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
May 20 16:27:53.903330 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.903305    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"]
May 20 16:27:53.916435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.916412    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcx2\" (UniqueName: \"kubernetes.io/projected/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kube-api-access-qmcx2\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:53.916548 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.916452    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:53.916548 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.916520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:53.916660 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:53.916572    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.017411 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.017384    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcx2\" (UniqueName: \"kubernetes.io/projected/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kube-api-access-qmcx2\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.017532 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.017434    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.017532 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.017491    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.017641 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.017547    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.017641 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:27:54.017613    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
May 20 16:27:54.017752 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:27:54.017694    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls podName:6d4d7dd4-80ef-4e88-ae4b-f67284a00624 nodeName:}" failed. No retries permitted until 2026-05-20 16:27:54.517680118 +0000 UTC m=+3317.208936027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls") pod "isvc-xgboost-runtime-predictor-779db84d9-j8q85" (UID: "6d4d7dd4-80ef-4e88-ae4b-f67284a00624") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
May 20 16:27:54.017940 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.017921    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.018300 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.018277    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.035817 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.035790    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcx2\" (UniqueName: \"kubernetes.io/projected/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kube-api-access-qmcx2\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.333769 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.333740    2556 generic.go:358] "Generic (PLEG): container finished" podID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerID="0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71" exitCode=2
May 20 16:27:54.333924 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.333815    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" event={"ID":"a0ef788c-dd66-46c1-b959-58b7193d8bc0","Type":"ContainerDied","Data":"0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71"}
May 20 16:27:54.521238 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.521202    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.523559 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.523539    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-j8q85\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.802048 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.802019    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:27:54.925918 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:54.925890    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"]
May 20 16:27:54.928421 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:27:54.928395    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4d7dd4_80ef_4e88_ae4b_f67284a00624.slice/crio-40ff6cb4e90c91c00ceb8540d6d1e502acc615c7003621cbb0b9b9e7e2f4fb02 WatchSource:0}: Error finding container 40ff6cb4e90c91c00ceb8540d6d1e502acc615c7003621cbb0b9b9e7e2f4fb02: Status 404 returned error can't find the container with id 40ff6cb4e90c91c00ceb8540d6d1e502acc615c7003621cbb0b9b9e7e2f4fb02
May 20 16:27:55.337532 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:55.337492    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" event={"ID":"6d4d7dd4-80ef-4e88-ae4b-f67284a00624","Type":"ContainerStarted","Data":"1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2"}
May 20 16:27:55.337699 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:55.337535    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" event={"ID":"6d4d7dd4-80ef-4e88-ae4b-f67284a00624","Type":"ContainerStarted","Data":"40ff6cb4e90c91c00ceb8540d6d1e502acc615c7003621cbb0b9b9e7e2f4fb02"}
May 20 16:27:56.218064 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:56.217972    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
May 20 16:27:56.224462 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:56.224431    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
May 20 16:27:59.165451 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.165421    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:59.252763 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.252737    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ef788c-dd66-46c1-b959-58b7193d8bc0-proxy-tls\") pod \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") "
May 20 16:27:59.252875 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.252770    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l7kj\" (UniqueName: \"kubernetes.io/projected/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kube-api-access-5l7kj\") pod \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") "
May 20 16:27:59.252875 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.252796    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ef788c-dd66-46c1-b959-58b7193d8bc0-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") "
May 20 16:27:59.252875 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.252847    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kserve-provision-location\") pod \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\" (UID: \"a0ef788c-dd66-46c1-b959-58b7193d8bc0\") "
May 20 16:27:59.253192 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.253169    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ef788c-dd66-46c1-b959-58b7193d8bc0-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "a0ef788c-dd66-46c1-b959-58b7193d8bc0" (UID: "a0ef788c-dd66-46c1-b959-58b7193d8bc0"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:27:59.253263 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.253211    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0ef788c-dd66-46c1-b959-58b7193d8bc0" (UID: "a0ef788c-dd66-46c1-b959-58b7193d8bc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:27:59.254792 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.254769    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ef788c-dd66-46c1-b959-58b7193d8bc0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0ef788c-dd66-46c1-b959-58b7193d8bc0" (UID: "a0ef788c-dd66-46c1-b959-58b7193d8bc0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:27:59.254884 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.254818    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kube-api-access-5l7kj" (OuterVolumeSpecName: "kube-api-access-5l7kj") pod "a0ef788c-dd66-46c1-b959-58b7193d8bc0" (UID: "a0ef788c-dd66-46c1-b959-58b7193d8bc0"). InnerVolumeSpecName "kube-api-access-5l7kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:27:59.348625 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.348598    2556 generic.go:358] "Generic (PLEG): container finished" podID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerID="19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed" exitCode=0
May 20 16:27:59.348727 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.348692    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"
May 20 16:27:59.348782 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.348690    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" event={"ID":"a0ef788c-dd66-46c1-b959-58b7193d8bc0","Type":"ContainerDied","Data":"19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed"}
May 20 16:27:59.348831 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.348809    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7" event={"ID":"a0ef788c-dd66-46c1-b959-58b7193d8bc0","Type":"ContainerDied","Data":"f1b30cc8aa5448fe89ffe8d0310fc8ae334250df09cb7bf24fa0ad05abb5d373"}
May 20 16:27:59.348886 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.348834    2556 scope.go:117] "RemoveContainer" containerID="0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71"
May 20 16:27:59.350163 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.350136    2556 generic.go:358] "Generic (PLEG): container finished" podID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerID="1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2" exitCode=0
May 20 16:27:59.350273 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.350177    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" event={"ID":"6d4d7dd4-80ef-4e88-ae4b-f67284a00624","Type":"ContainerDied","Data":"1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2"}
May 20 16:27:59.353733 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.353709    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ef788c-dd66-46c1-b959-58b7193d8bc0-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:27:59.353733 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.353734    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l7kj\" (UniqueName: \"kubernetes.io/projected/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kube-api-access-5l7kj\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:27:59.353870 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.353746    2556 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ef788c-dd66-46c1-b959-58b7193d8bc0-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:27:59.353870 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.353756    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ef788c-dd66-46c1-b959-58b7193d8bc0-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:27:59.356598 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.356578    2556 scope.go:117] "RemoveContainer" containerID="19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed"
May 20 16:27:59.363833 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.363772    2556 scope.go:117] "RemoveContainer" containerID="4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d"
May 20 16:27:59.371155 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.371134    2556 scope.go:117] "RemoveContainer" containerID="0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71"
May 20 16:27:59.371476 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:27:59.371449    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71\": container with ID starting with 0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71 not found: ID does not exist" containerID="0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71"
May 20 16:27:59.371548 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.371486    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71"} err="failed to get container status \"0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71\": rpc error: code = NotFound desc = could not find container \"0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71\": container with ID starting with 0a24b0130286879ba23ca05eae1785ef80f32939849611e5993952c008044b71 not found: ID does not exist"
May 20 16:27:59.371548 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.371511    2556 scope.go:117] "RemoveContainer" containerID="19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed"
May 20 16:27:59.371785 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:27:59.371768    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed\": container with ID starting with 19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed not found: ID does not exist" containerID="19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed"
May 20 16:27:59.371837 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.371791    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed"} err="failed to get container status \"19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed\": rpc error: code = NotFound desc = could not find container \"19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed\": container with ID starting with 19ad741f6f63f3ba989509868afa801ff2fcd99c953f0caa07446972ba00d8ed not found: ID does not exist"
May 20 16:27:59.371837 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.371807    2556 scope.go:117] "RemoveContainer" containerID="4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d"
May 20 16:27:59.372017 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:27:59.372003    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d\": container with ID starting with 4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d not found: ID does not exist" containerID="4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d"
May 20 16:27:59.372062 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.372020    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d"} err="failed to get container status \"4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d\": rpc error: code = NotFound desc = could not find container \"4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d\": container with ID starting with 4c15e52b69d7d888c6d4f1c9dd5b9c4cad8b5f6fb95bf5ad9f87e00c89d53e4d not found: ID does not exist"
May 20 16:27:59.386277 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.386252    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"]
May 20 16:27:59.390266 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.390237    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-25wv7"]
May 20 16:27:59.890631 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:27:59.890589    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" path="/var/lib/kubelet/pods/a0ef788c-dd66-46c1-b959-58b7193d8bc0/volumes"
May 20 16:28:00.355314 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:00.355278    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" event={"ID":"6d4d7dd4-80ef-4e88-ae4b-f67284a00624","Type":"ContainerStarted","Data":"67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93"}
May 20 16:28:00.355757 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:00.355331    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" event={"ID":"6d4d7dd4-80ef-4e88-ae4b-f67284a00624","Type":"ContainerStarted","Data":"0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7"}
May 20 16:28:00.355757 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:00.355578    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:28:00.375467 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:00.375426    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podStartSLOduration=7.375412037 podStartE2EDuration="7.375412037s" podCreationTimestamp="2026-05-20 16:27:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:28:00.375141906 +0000 UTC m=+3323.066397834" watchObservedRunningTime="2026-05-20 16:28:00.375412037 +0000 UTC m=+3323.066667965"
May 20 16:28:01.359808 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:01.359778    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:28:01.361175 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:01.361145    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
May 20 16:28:02.362491 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:02.362451    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
May 20 16:28:07.366585 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:07.366555    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:28:07.367125 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:07.367077    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
May 20 16:28:17.367309 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:17.367269    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
May 20 16:28:27.367830 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:27.367794    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
May 20 16:28:37.367221 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:37.367176    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
May 20 16:28:47.367187 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:47.367146    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
May 20 16:28:57.367919 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:28:57.367891    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:29:03.972615 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:03.972580    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"]
May 20 16:29:03.973142 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:03.972944    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container" containerID="cri-o://0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7" gracePeriod=30
May 20 16:29:03.973142 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:03.972992    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kube-rbac-proxy" containerID="cri-o://67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93" gracePeriod=30
May 20 16:29:04.038869 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.038841    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"]
May 20 16:29:04.039150 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.039137    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="storage-initializer"
May 20 16:29:04.039199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.039152    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="storage-initializer"
May 20 16:29:04.039199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.039159    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kserve-container"
May 20 16:29:04.039199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.039165    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kserve-container"
May 20 16:29:04.039199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.039171    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kube-rbac-proxy"
May 20 16:29:04.039199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.039176    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kube-rbac-proxy"
May 20 16:29:04.039355 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.039222    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kube-rbac-proxy"
May 20 16:29:04.039355 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.039230    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0ef788c-dd66-46c1-b959-58b7193d8bc0" containerName="kserve-container"
May 20 16:29:04.042124 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.042094    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.044350 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.044330    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
May 20 16:29:04.044420 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.044334    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
May 20 16:29:04.052424 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.052404    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"]
May 20 16:29:04.191219 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.191191    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0aef7e-8438-4251-9f1c-363c16173ffe-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.191333 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.191226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ttl\" (UniqueName: \"kubernetes.io/projected/3f0aef7e-8438-4251-9f1c-363c16173ffe-kube-api-access-c2ttl\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.191333 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.191280    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0aef7e-8438-4251-9f1c-363c16173ffe-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.191333 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.191315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.292385 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.292362    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0aef7e-8438-4251-9f1c-363c16173ffe-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.292491 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.292393    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ttl\" (UniqueName: \"kubernetes.io/projected/3f0aef7e-8438-4251-9f1c-363c16173ffe-kube-api-access-c2ttl\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.292491 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.292427    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0aef7e-8438-4251-9f1c-363c16173ffe-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.292491 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.292480    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.292629 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:29:04.292615    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
May 20 16:29:04.292703 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.292686    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0aef7e-8438-4251-9f1c-363c16173ffe-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.292742 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:29:04.292706    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls podName:3f0aef7e-8438-4251-9f1c-363c16173ffe nodeName:}" failed. No retries permitted until 2026-05-20 16:29:04.792685093 +0000 UTC m=+3387.483941014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" (UID: "3f0aef7e-8438-4251-9f1c-363c16173ffe") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
May 20 16:29:04.293198 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.293182    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0aef7e-8438-4251-9f1c-363c16173ffe-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.303360 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.303338    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ttl\" (UniqueName: \"kubernetes.io/projected/3f0aef7e-8438-4251-9f1c-363c16173ffe-kube-api-access-c2ttl\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.538385 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.538350    2556 generic.go:358] "Generic (PLEG): container finished" podID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerID="67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93" exitCode=2
May 20 16:29:04.538542 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.538426    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" event={"ID":"6d4d7dd4-80ef-4e88-ae4b-f67284a00624","Type":"ContainerDied","Data":"67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93"}
May 20 16:29:04.797305 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.797277    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.799623 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.799601    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:04.951329 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:04.951290    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:05.074956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:05.074879    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"]
May 20 16:29:05.077886 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:29:05.077859    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0aef7e_8438_4251_9f1c_363c16173ffe.slice/crio-6dc8bea3e4b7654ca9d904e8290f887762a6645f3d47c4699cf8b5fa8cf388e3 WatchSource:0}: Error finding container 6dc8bea3e4b7654ca9d904e8290f887762a6645f3d47c4699cf8b5fa8cf388e3: Status 404 returned error can't find the container with id 6dc8bea3e4b7654ca9d904e8290f887762a6645f3d47c4699cf8b5fa8cf388e3
May 20 16:29:05.542427 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:05.542393    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" event={"ID":"3f0aef7e-8438-4251-9f1c-363c16173ffe","Type":"ContainerStarted","Data":"bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11"}
May 20 16:29:05.542601 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:05.542431    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" event={"ID":"3f0aef7e-8438-4251-9f1c-363c16173ffe","Type":"ContainerStarted","Data":"6dc8bea3e4b7654ca9d904e8290f887762a6645f3d47c4699cf8b5fa8cf388e3"}
May 20 16:29:07.218071 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.218046    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:29:07.316539 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.316478    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") "
May 20 16:29:07.316539 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.316517    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmcx2\" (UniqueName: \"kubernetes.io/projected/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kube-api-access-qmcx2\") pod \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") "
May 20 16:29:07.316715 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.316560    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kserve-provision-location\") pod \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") "
May 20 16:29:07.316715 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.316585    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls\") pod \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\" (UID: \"6d4d7dd4-80ef-4e88-ae4b-f67284a00624\") "
May 20 16:29:07.316864 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.316840    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "6d4d7dd4-80ef-4e88-ae4b-f67284a00624" (UID: "6d4d7dd4-80ef-4e88-ae4b-f67284a00624"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:29:07.316932 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.316872    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d4d7dd4-80ef-4e88-ae4b-f67284a00624" (UID: "6d4d7dd4-80ef-4e88-ae4b-f67284a00624"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:29:07.318605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.318586    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kube-api-access-qmcx2" (OuterVolumeSpecName: "kube-api-access-qmcx2") pod "6d4d7dd4-80ef-4e88-ae4b-f67284a00624" (UID: "6d4d7dd4-80ef-4e88-ae4b-f67284a00624"). InnerVolumeSpecName "kube-api-access-qmcx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:29:07.318678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.318660    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d4d7dd4-80ef-4e88-ae4b-f67284a00624" (UID: "6d4d7dd4-80ef-4e88-ae4b-f67284a00624"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:29:07.417253 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.417229    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:29:07.417253 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.417250    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:29:07.417376 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.417260    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:29:07.417376 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.417270    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmcx2\" (UniqueName: \"kubernetes.io/projected/6d4d7dd4-80ef-4e88-ae4b-f67284a00624-kube-api-access-qmcx2\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:29:07.552257 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.552222    2556 generic.go:358] "Generic (PLEG): container finished" podID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerID="0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7" exitCode=0
May 20 16:29:07.552359 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.552309    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"
May 20 16:29:07.552359 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.552319    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" event={"ID":"6d4d7dd4-80ef-4e88-ae4b-f67284a00624","Type":"ContainerDied","Data":"0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7"}
May 20 16:29:07.552449 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.552365    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85" event={"ID":"6d4d7dd4-80ef-4e88-ae4b-f67284a00624","Type":"ContainerDied","Data":"40ff6cb4e90c91c00ceb8540d6d1e502acc615c7003621cbb0b9b9e7e2f4fb02"}
May 20 16:29:07.552449 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.552383    2556 scope.go:117] "RemoveContainer" containerID="67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93"
May 20 16:29:07.561674 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.561515    2556 scope.go:117] "RemoveContainer" containerID="0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7"
May 20 16:29:07.568891 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.568866    2556 scope.go:117] "RemoveContainer" containerID="1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2"
May 20 16:29:07.575398 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.575371    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"]
May 20 16:29:07.576161 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.576136    2556 scope.go:117] "RemoveContainer" containerID="67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93"
May 20 16:29:07.576403 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:29:07.576385    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93\": container with ID starting with 67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93 not found: ID does not exist" containerID="67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93"
May 20 16:29:07.576460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.576410    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93"} err="failed to get container status \"67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93\": rpc error: code = NotFound desc = could not find container \"67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93\": container with ID starting with 67b2e69179dac316fbebf3821c9dd468bcdb027cc62464798b74b90b2265db93 not found: ID does not exist"
May 20 16:29:07.576460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.576427    2556 scope.go:117] "RemoveContainer" containerID="0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7"
May 20 16:29:07.576699 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:29:07.576677    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7\": container with ID starting with 0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7 not found: ID does not exist" containerID="0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7"
May 20 16:29:07.576749 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.576702    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7"} err="failed to get container status \"0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7\": rpc error: code = NotFound desc = could not find container \"0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7\": container with ID starting with 0380e65be453c6a690f6fc477994271857499162b8f0fbf7b1c008c4bec58cf7 not found: ID does not exist"
May 20 16:29:07.576749 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.576718    2556 scope.go:117] "RemoveContainer" containerID="1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2"
May 20 16:29:07.576967 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:29:07.576952    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2\": container with ID starting with 1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2 not found: ID does not exist" containerID="1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2"
May 20 16:29:07.577017 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.576969    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2"} err="failed to get container status \"1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2\": rpc error: code = NotFound desc = could not find container \"1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2\": container with ID starting with 1fd72b372c266321640faae9dbb7fe247df94d4dab4830bf19d7fa8184d227b2 not found: ID does not exist"
May 20 16:29:07.579778 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.579753    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-j8q85"]
May 20 16:29:07.892463 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:07.892401    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" path="/var/lib/kubelet/pods/6d4d7dd4-80ef-4e88-ae4b-f67284a00624/volumes"
May 20 16:29:09.560478 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:09.560440    2556 generic.go:358] "Generic (PLEG): container finished" podID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerID="bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11" exitCode=0
May 20 16:29:09.560873 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:09.560494    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" event={"ID":"3f0aef7e-8438-4251-9f1c-363c16173ffe","Type":"ContainerDied","Data":"bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11"}
May 20 16:29:10.565517 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:10.565479    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" event={"ID":"3f0aef7e-8438-4251-9f1c-363c16173ffe","Type":"ContainerStarted","Data":"6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83"}
May 20 16:29:10.565517 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:10.565523    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" event={"ID":"3f0aef7e-8438-4251-9f1c-363c16173ffe","Type":"ContainerStarted","Data":"2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194"}
May 20 16:29:10.565991 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:10.565856    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:10.565991 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:10.565890    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:10.585380 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:10.585325    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" podStartSLOduration=6.585307274 podStartE2EDuration="6.585307274s" podCreationTimestamp="2026-05-20 16:29:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:29:10.584557185 +0000 UTC m=+3393.275813113" watchObservedRunningTime="2026-05-20 16:29:10.585307274 +0000 UTC m=+3393.276563203"
May 20 16:29:16.575076 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:16.575045    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:29:46.604670 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:46.604569    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 20 16:29:56.577890 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:29:56.577861    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:30:04.164329 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.164292    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"]
May 20 16:30:04.164806 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.164758    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kserve-container" containerID="cri-o://2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194" gracePeriod=30
May 20 16:30:04.164900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.164762    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kube-rbac-proxy" containerID="cri-o://6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83" gracePeriod=30
May 20 16:30:04.242767 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.242737    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"]
May 20 16:30:04.243115 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.243079    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kube-rbac-proxy"
May 20 16:30:04.243115 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.243112    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kube-rbac-proxy"
May 20 16:30:04.243264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.243127    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container"
May 20 16:30:04.243264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.243135    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container"
May 20 16:30:04.243264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.243154    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="storage-initializer"
May 20 16:30:04.243264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.243164    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="storage-initializer"
May 20 16:30:04.243264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.243226    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kube-rbac-proxy"
May 20 16:30:04.243264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.243243    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="6d4d7dd4-80ef-4e88-ae4b-f67284a00624" containerName="kserve-container"
May 20 16:30:04.246312 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.246292    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.248607 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.248587    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
May 20 16:30:04.248708 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.248591    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
May 20 16:30:04.259485 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.259460    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"]
May 20 16:30:04.310603 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.310563    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12369b39-45e0-416b-bfcc-281c78a1761c-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.310603 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.310607    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12369b39-45e0-416b-bfcc-281c78a1761c-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.310828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.310694    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.310828 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.310727    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977qn\" (UniqueName: \"kubernetes.io/projected/12369b39-45e0-416b-bfcc-281c78a1761c-kube-api-access-977qn\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.412069 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.412033    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.412069 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.412073    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-977qn\" (UniqueName: \"kubernetes.io/projected/12369b39-45e0-416b-bfcc-281c78a1761c-kube-api-access-977qn\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.412287 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.412141    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12369b39-45e0-416b-bfcc-281c78a1761c-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.412287 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.412168    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12369b39-45e0-416b-bfcc-281c78a1761c-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.412287 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:30:04.412224    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
May 20 16:30:04.412471 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:30:04.412303    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls podName:12369b39-45e0-416b-bfcc-281c78a1761c nodeName:}" failed. No retries permitted until 2026-05-20 16:30:04.912279161 +0000 UTC m=+3447.603535068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls") pod "isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" (UID: "12369b39-45e0-416b-bfcc-281c78a1761c") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
May 20 16:30:04.412565 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.412505    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12369b39-45e0-416b-bfcc-281c78a1761c-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.412868 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.412844    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12369b39-45e0-416b-bfcc-281c78a1761c-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.422629 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.422571    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-977qn\" (UniqueName: \"kubernetes.io/projected/12369b39-45e0-416b-bfcc-281c78a1761c-kube-api-access-977qn\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.716719 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.716635    2556 generic.go:358] "Generic (PLEG): container finished" podID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerID="6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83" exitCode=2
May 20 16:30:04.716719 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.716706    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" event={"ID":"3f0aef7e-8438-4251-9f1c-363c16173ffe","Type":"ContainerDied","Data":"6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83"}
May 20 16:30:04.916063 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.916016    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:04.918394 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:04.918370    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-cck9s\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:05.155435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:05.155394    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:05.311770 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:05.311733    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"]
May 20 16:30:05.315509 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:30:05.315477    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12369b39_45e0_416b_bfcc_281c78a1761c.slice/crio-466d46bbf20a26e44ca6bab5ed11744a6ee8f17c8ee6e28809d06d0f5620fa9e WatchSource:0}: Error finding container 466d46bbf20a26e44ca6bab5ed11744a6ee8f17c8ee6e28809d06d0f5620fa9e: Status 404 returned error can't find the container with id 466d46bbf20a26e44ca6bab5ed11744a6ee8f17c8ee6e28809d06d0f5620fa9e
May 20 16:30:05.720267 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:05.720229    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" event={"ID":"12369b39-45e0-416b-bfcc-281c78a1761c","Type":"ContainerStarted","Data":"43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68"}
May 20 16:30:05.720267 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:05.720266    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" event={"ID":"12369b39-45e0-416b-bfcc-281c78a1761c","Type":"ContainerStarted","Data":"466d46bbf20a26e44ca6bab5ed11744a6ee8f17c8ee6e28809d06d0f5620fa9e"}
May 20 16:30:06.569621 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:06.569578    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
May 20 16:30:09.732361 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:09.732269    2556 generic.go:358] "Generic (PLEG): container finished" podID="12369b39-45e0-416b-bfcc-281c78a1761c" containerID="43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68" exitCode=0
May 20 16:30:09.732361 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:09.732340    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" event={"ID":"12369b39-45e0-416b-bfcc-281c78a1761c","Type":"ContainerDied","Data":"43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68"}
May 20 16:30:10.736790 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:10.736754    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" event={"ID":"12369b39-45e0-416b-bfcc-281c78a1761c","Type":"ContainerStarted","Data":"c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67"}
May 20 16:30:10.736790 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:10.736797    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" event={"ID":"12369b39-45e0-416b-bfcc-281c78a1761c","Type":"ContainerStarted","Data":"001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c"}
May 20 16:30:10.737308 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:10.737094    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:10.737308 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:10.737239    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:10.738347 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:10.738321    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
May 20 16:30:10.757416 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:10.757381    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podStartSLOduration=6.75737002 podStartE2EDuration="6.75737002s" podCreationTimestamp="2026-05-20 16:30:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:30:10.755677006 +0000 UTC m=+3453.446932935" watchObservedRunningTime="2026-05-20 16:30:10.75737002 +0000 UTC m=+3453.448625947"
May 20 16:30:11.093260 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.093237    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:30:11.167220 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.167192    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0aef7e-8438-4251-9f1c-363c16173ffe-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"3f0aef7e-8438-4251-9f1c-363c16173ffe\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") "
May 20 16:30:11.167383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.167234    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2ttl\" (UniqueName: \"kubernetes.io/projected/3f0aef7e-8438-4251-9f1c-363c16173ffe-kube-api-access-c2ttl\") pod \"3f0aef7e-8438-4251-9f1c-363c16173ffe\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") "
May 20 16:30:11.167383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.167264    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls\") pod \"3f0aef7e-8438-4251-9f1c-363c16173ffe\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") "
May 20 16:30:11.167514 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.167378    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0aef7e-8438-4251-9f1c-363c16173ffe-kserve-provision-location\") pod \"3f0aef7e-8438-4251-9f1c-363c16173ffe\" (UID: \"3f0aef7e-8438-4251-9f1c-363c16173ffe\") "
May 20 16:30:11.167593 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.167570    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0aef7e-8438-4251-9f1c-363c16173ffe-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "3f0aef7e-8438-4251-9f1c-363c16173ffe" (UID: "3f0aef7e-8438-4251-9f1c-363c16173ffe"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:30:11.167646 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.167629    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0aef7e-8438-4251-9f1c-363c16173ffe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f0aef7e-8438-4251-9f1c-363c16173ffe" (UID: "3f0aef7e-8438-4251-9f1c-363c16173ffe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:30:11.169326 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.169301    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0aef7e-8438-4251-9f1c-363c16173ffe-kube-api-access-c2ttl" (OuterVolumeSpecName: "kube-api-access-c2ttl") pod "3f0aef7e-8438-4251-9f1c-363c16173ffe" (UID: "3f0aef7e-8438-4251-9f1c-363c16173ffe"). InnerVolumeSpecName "kube-api-access-c2ttl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:30:11.169410 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.169324    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f0aef7e-8438-4251-9f1c-363c16173ffe" (UID: "3f0aef7e-8438-4251-9f1c-363c16173ffe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:30:11.268564 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.268540    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0aef7e-8438-4251-9f1c-363c16173ffe-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:30:11.268564 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.268560    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2ttl\" (UniqueName: \"kubernetes.io/projected/3f0aef7e-8438-4251-9f1c-363c16173ffe-kube-api-access-c2ttl\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:30:11.268718 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.268571    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0aef7e-8438-4251-9f1c-363c16173ffe-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:30:11.268718 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.268580    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0aef7e-8438-4251-9f1c-363c16173ffe-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:30:11.740899 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.740862    2556 generic.go:358] "Generic (PLEG): container finished" podID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerID="2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194" exitCode=0
May 20 16:30:11.741381 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.740937    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"
May 20 16:30:11.741381 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.740950    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" event={"ID":"3f0aef7e-8438-4251-9f1c-363c16173ffe","Type":"ContainerDied","Data":"2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194"}
May 20 16:30:11.741381 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.740994    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9" event={"ID":"3f0aef7e-8438-4251-9f1c-363c16173ffe","Type":"ContainerDied","Data":"6dc8bea3e4b7654ca9d904e8290f887762a6645f3d47c4699cf8b5fa8cf388e3"}
May 20 16:30:11.741381 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.741017    2556 scope.go:117] "RemoveContainer" containerID="6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83"
May 20 16:30:11.741668 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.741643    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
May 20 16:30:11.749335 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.749294    2556 scope.go:117] "RemoveContainer" containerID="2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194"
May 20 16:30:11.755952 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.755936    2556 scope.go:117] "RemoveContainer" containerID="bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11"
May 20 16:30:11.762616 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.762595    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"]
May 20 16:30:11.763566 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.763545    2556 scope.go:117] "RemoveContainer" containerID="6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83"
May 20 16:30:11.763811 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:30:11.763791    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83\": container with ID starting with 6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83 not found: ID does not exist" containerID="6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83"
May 20 16:30:11.763942 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.763822    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83"} err="failed to get container status \"6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83\": rpc error: code = NotFound desc = could not find container \"6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83\": container with ID starting with 6accb74b0bdc8f801d9f20b9524b37a902e5a5462d10ed1f2606a834a9689f83 not found: ID does not exist"
May 20 16:30:11.763942 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.763845    2556 scope.go:117] "RemoveContainer" containerID="2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194"
May 20 16:30:11.764192 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:30:11.764168    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194\": container with ID starting with 2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194 not found: ID does not exist" containerID="2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194"
May 20 16:30:11.764264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.764202    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194"} err="failed to get container status \"2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194\": rpc error: code = NotFound desc = could not find container \"2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194\": container with ID starting with 2175dbbce53a15360f619500445a3276bfa3929dcd13e45d96e8b89cb765a194 not found: ID does not exist"
May 20 16:30:11.764264 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.764218    2556 scope.go:117] "RemoveContainer" containerID="bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11"
May 20 16:30:11.764599 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:30:11.764495    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11\": container with ID starting with bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11 not found: ID does not exist" containerID="bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11"
May 20 16:30:11.764599 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.764526    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11"} err="failed to get container status \"bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11\": rpc error: code = NotFound desc = could not find container \"bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11\": container with ID starting with bfd1aa10e0c89f26ea3dec14c440f980018411133c4e6d16a8f7fccd6724ec11 not found: ID does not exist"
May 20 16:30:11.766021 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.765994    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sxnf9"]
May 20 16:30:11.891312 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:11.891284    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" path="/var/lib/kubelet/pods/3f0aef7e-8438-4251-9f1c-363c16173ffe/volumes"
May 20 16:30:16.745321 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:16.745293    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:30:16.745853 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:16.745827    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
May 20 16:30:26.746351 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:26.746307    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
May 20 16:30:36.746218 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:36.746178    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
May 20 16:30:46.746594 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:46.746554    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
May 20 16:30:56.745859 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:30:56.745767    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
May 20 16:31:06.746276 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:06.746250    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:31:14.365004 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.364970    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"]
May 20 16:31:14.365453 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.365280    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" containerID="cri-o://001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c" gracePeriod=30
May 20 16:31:14.365453 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.365345    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kube-rbac-proxy" containerID="cri-o://c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67" gracePeriod=30
May 20 16:31:14.445193 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445169    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"]
May 20 16:31:14.445417 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445406    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kserve-container"
May 20 16:31:14.445473 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445419    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kserve-container"
May 20 16:31:14.445473 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445431    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kube-rbac-proxy"
May 20 16:31:14.445473 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445436    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kube-rbac-proxy"
May 20 16:31:14.445473 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445455    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="storage-initializer"
May 20 16:31:14.445473 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445461    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="storage-initializer"
May 20 16:31:14.445630 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445499    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kserve-container"
May 20 16:31:14.445630 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.445509    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f0aef7e-8438-4251-9f1c-363c16173ffe" containerName="kube-rbac-proxy"
May 20 16:31:14.448542 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.448526    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.451032 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.451013    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
May 20 16:31:14.451032 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.451027    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
May 20 16:31:14.451206 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.451033    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 20 16:31:14.458843 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.458823    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"]
May 20 16:31:14.612409 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.612378    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zskr\" (UniqueName: \"kubernetes.io/projected/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kube-api-access-4zskr\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.612590 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.612440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.612590 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.612519    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.612590 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.612577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.713675 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.713594    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.713675 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.713645    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.713675 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.713677    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4zskr\" (UniqueName: \"kubernetes.io/projected/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kube-api-access-4zskr\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.713936 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.713706    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.714044 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.714021    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.714322 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.714304    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.715961 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.715944    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.725354 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.725335    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zskr\" (UniqueName: \"kubernetes.io/projected/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kube-api-access-4zskr\") pod \"isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.758140 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.758119    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:14.881042 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.881015    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"]
May 20 16:31:14.883518 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:31:14.883491    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542f44d0_1bb3_49f8_8881_6fe1c2fe6805.slice/crio-2380a58181496579911d1de2dcc0da80859c39da72f07021f311802fe72e0866 WatchSource:0}: Error finding container 2380a58181496579911d1de2dcc0da80859c39da72f07021f311802fe72e0866: Status 404 returned error can't find the container with id 2380a58181496579911d1de2dcc0da80859c39da72f07021f311802fe72e0866
May 20 16:31:14.885312 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.885295    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:31:14.917309 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.917279    2556 generic.go:358] "Generic (PLEG): container finished" podID="12369b39-45e0-416b-bfcc-281c78a1761c" containerID="c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67" exitCode=2
May 20 16:31:14.917408 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.917348    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" event={"ID":"12369b39-45e0-416b-bfcc-281c78a1761c","Type":"ContainerDied","Data":"c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67"}
May 20 16:31:14.918411 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:14.918391    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" event={"ID":"542f44d0-1bb3-49f8-8881-6fe1c2fe6805","Type":"ContainerStarted","Data":"2380a58181496579911d1de2dcc0da80859c39da72f07021f311802fe72e0866"}
May 20 16:31:15.922470 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:15.922428    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" event={"ID":"542f44d0-1bb3-49f8-8881-6fe1c2fe6805","Type":"ContainerStarted","Data":"8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2"}
May 20 16:31:16.742196 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:16.742157    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
May 20 16:31:16.746403 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:16.746382    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
May 20 16:31:16.926003 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:16.925969    2556 generic.go:358] "Generic (PLEG): container finished" podID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerID="8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2" exitCode=0
May 20 16:31:16.926500 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:16.926056    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" event={"ID":"542f44d0-1bb3-49f8-8881-6fe1c2fe6805","Type":"ContainerDied","Data":"8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2"}
May 20 16:31:17.801789 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.801768    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:31:17.930353 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.930321    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" event={"ID":"542f44d0-1bb3-49f8-8881-6fe1c2fe6805","Type":"ContainerStarted","Data":"47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2"}
May 20 16:31:17.930719 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.930355    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" event={"ID":"542f44d0-1bb3-49f8-8881-6fe1c2fe6805","Type":"ContainerStarted","Data":"418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38"}
May 20 16:31:17.930719 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.930511    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:17.931943 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.931924    2556 generic.go:358] "Generic (PLEG): container finished" podID="12369b39-45e0-416b-bfcc-281c78a1761c" containerID="001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c" exitCode=0
May 20 16:31:17.932040 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.931964    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" event={"ID":"12369b39-45e0-416b-bfcc-281c78a1761c","Type":"ContainerDied","Data":"001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c"}
May 20 16:31:17.932040 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.931982    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s" event={"ID":"12369b39-45e0-416b-bfcc-281c78a1761c","Type":"ContainerDied","Data":"466d46bbf20a26e44ca6bab5ed11744a6ee8f17c8ee6e28809d06d0f5620fa9e"}
May 20 16:31:17.932040 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.931987    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"
May 20 16:31:17.932040 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.931996    2556 scope.go:117] "RemoveContainer" containerID="c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67"
May 20 16:31:17.939061 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.939046    2556 scope.go:117] "RemoveContainer" containerID="001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c"
May 20 16:31:17.941996 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.941980    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12369b39-45e0-416b-bfcc-281c78a1761c-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"12369b39-45e0-416b-bfcc-281c78a1761c\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") "
May 20 16:31:17.942058 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.942016    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls\") pod \"12369b39-45e0-416b-bfcc-281c78a1761c\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") "
May 20 16:31:17.942146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.942058    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12369b39-45e0-416b-bfcc-281c78a1761c-kserve-provision-location\") pod \"12369b39-45e0-416b-bfcc-281c78a1761c\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") "
May 20 16:31:17.942146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.942121    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-977qn\" (UniqueName: \"kubernetes.io/projected/12369b39-45e0-416b-bfcc-281c78a1761c-kube-api-access-977qn\") pod \"12369b39-45e0-416b-bfcc-281c78a1761c\" (UID: \"12369b39-45e0-416b-bfcc-281c78a1761c\") "
May 20 16:31:17.942360 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.942338    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12369b39-45e0-416b-bfcc-281c78a1761c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12369b39-45e0-416b-bfcc-281c78a1761c" (UID: "12369b39-45e0-416b-bfcc-281c78a1761c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:31:17.942421 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.942403    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12369b39-45e0-416b-bfcc-281c78a1761c-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "12369b39-45e0-416b-bfcc-281c78a1761c" (UID: "12369b39-45e0-416b-bfcc-281c78a1761c"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:31:17.944037 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.944013    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12369b39-45e0-416b-bfcc-281c78a1761c" (UID: "12369b39-45e0-416b-bfcc-281c78a1761c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:31:17.944239 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.944214    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12369b39-45e0-416b-bfcc-281c78a1761c-kube-api-access-977qn" (OuterVolumeSpecName: "kube-api-access-977qn") pod "12369b39-45e0-416b-bfcc-281c78a1761c" (UID: "12369b39-45e0-416b-bfcc-281c78a1761c"). InnerVolumeSpecName "kube-api-access-977qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:31:17.947146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.947129    2556 scope.go:117] "RemoveContainer" containerID="43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68"
May 20 16:31:17.949615 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.949572    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podStartSLOduration=3.949562151 podStartE2EDuration="3.949562151s" podCreationTimestamp="2026-05-20 16:31:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:31:17.94790543 +0000 UTC m=+3520.639161359" watchObservedRunningTime="2026-05-20 16:31:17.949562151 +0000 UTC m=+3520.640818080"
May 20 16:31:17.957497 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.957479    2556 scope.go:117] "RemoveContainer" containerID="c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67"
May 20 16:31:17.957776 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:31:17.957758    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67\": container with ID starting with c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67 not found: ID does not exist" containerID="c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67"
May 20 16:31:17.957831 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.957783    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67"} err="failed to get container status \"c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67\": rpc error: code = NotFound desc = could not find container \"c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67\": container with ID starting with c0967dc4a4a8f3f3c67950ed14168a615e61b9005f150b1826d14621dee37b67 not found: ID does not exist"
May 20 16:31:17.957831 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.957803    2556 scope.go:117] "RemoveContainer" containerID="001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c"
May 20 16:31:17.958042 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:31:17.958022    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c\": container with ID starting with 001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c not found: ID does not exist" containerID="001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c"
May 20 16:31:17.958087 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.958048    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c"} err="failed to get container status \"001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c\": rpc error: code = NotFound desc = could not find container \"001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c\": container with ID starting with 001609d10e6bbfb5282e7b000bfd3ebede62d9d99976a07007b5a0832385f22c not found: ID does not exist"
May 20 16:31:17.958087 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.958064    2556 scope.go:117] "RemoveContainer" containerID="43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68"
May 20 16:31:17.958324 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:31:17.958306    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68\": container with ID starting with 43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68 not found: ID does not exist" containerID="43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68"
May 20 16:31:17.958375 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:17.958327    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68"} err="failed to get container status \"43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68\": rpc error: code = NotFound desc = could not find container \"43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68\": container with ID starting with 43177cfb0bd5ea3455d76f60701cebdc02faa5504828578725e9df05d5a0aa68 not found: ID does not exist"
May 20 16:31:18.043279 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:18.043245    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12369b39-45e0-416b-bfcc-281c78a1761c-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:31:18.043279 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:18.043279    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12369b39-45e0-416b-bfcc-281c78a1761c-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:31:18.043475 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:18.043296    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12369b39-45e0-416b-bfcc-281c78a1761c-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:31:18.043475 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:18.043311    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-977qn\" (UniqueName: \"kubernetes.io/projected/12369b39-45e0-416b-bfcc-281c78a1761c-kube-api-access-977qn\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:31:18.254951 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:18.254924    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"]
May 20 16:31:18.259707 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:18.259684    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-cck9s"]
May 20 16:31:18.935036 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:18.935007    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:18.936367 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:18.936339    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:31:19.890563 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:19.890532    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" path="/var/lib/kubelet/pods/12369b39-45e0-416b-bfcc-281c78a1761c/volumes"
May 20 16:31:19.937627 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:19.937599    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:31:24.942151 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:24.942120    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:31:24.942706 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:24.942679    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:31:34.942950 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:34.942914    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:31:44.943622 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:44.943582    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:31:54.942728 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:31:54.942691    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:32:04.942619 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:04.942576    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:32:14.943652 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:14.943614    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:32:24.944269 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:24.944235    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:32:34.559014 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.558978    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"]
May 20 16:32:34.559503 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.559396    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" containerID="cri-o://418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38" gracePeriod=30
May 20 16:32:34.559572 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.559490    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kube-rbac-proxy" containerID="cri-o://47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2" gracePeriod=30
May 20 16:32:34.713799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.713766    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"]
May 20 16:32:34.714033 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.714021    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kube-rbac-proxy"
May 20 16:32:34.714082 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.714034    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kube-rbac-proxy"
May 20 16:32:34.714082 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.714047    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container"
May 20 16:32:34.714082 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.714053    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container"
May 20 16:32:34.714082 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.714063    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="storage-initializer"
May 20 16:32:34.714082 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.714068    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="storage-initializer"
May 20 16:32:34.714263 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.714139    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kube-rbac-proxy"
May 20 16:32:34.714263 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.714150    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="12369b39-45e0-416b-bfcc-281c78a1761c" containerName="kserve-container"
May 20 16:32:34.719171 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.717686    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.720799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.720776    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
May 20 16:32:34.720799 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.720789    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
May 20 16:32:34.721025 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.720799    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 16:32:34.726422 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.726345    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"]
May 20 16:32:34.867146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.867047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/681da0df-e4f2-4a57-9863-d9914c41178f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.867146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.867092    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/681da0df-e4f2-4a57-9863-d9914c41178f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.867146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.867135    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.867374 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.867163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cq4s\" (UniqueName: \"kubernetes.io/projected/681da0df-e4f2-4a57-9863-d9914c41178f-kube-api-access-6cq4s\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.867374 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.867184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.938486 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.938442    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
May 20 16:32:34.942723 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.942696    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
May 20 16:32:34.968066 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.968035    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/681da0df-e4f2-4a57-9863-d9914c41178f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.968066 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.968066    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.968227 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.968094    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6cq4s\" (UniqueName: \"kubernetes.io/projected/681da0df-e4f2-4a57-9863-d9914c41178f-kube-api-access-6cq4s\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.968279 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.968225    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.968340 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.968309    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/681da0df-e4f2-4a57-9863-d9914c41178f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.968545 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.968522    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/681da0df-e4f2-4a57-9863-d9914c41178f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.968771 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.968750    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.968879 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.968798    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.970710 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.970690    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/681da0df-e4f2-4a57-9863-d9914c41178f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:34.976304 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:34.976281    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cq4s\" (UniqueName: \"kubernetes.io/projected/681da0df-e4f2-4a57-9863-d9914c41178f-kube-api-access-6cq4s\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:35.030108 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:35.030084    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:35.136708 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:35.136642    2556 generic.go:358] "Generic (PLEG): container finished" podID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerID="47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2" exitCode=2
May 20 16:32:35.136836 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:35.136711    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" event={"ID":"542f44d0-1bb3-49f8-8881-6fe1c2fe6805","Type":"ContainerDied","Data":"47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2"}
May 20 16:32:35.149657 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:35.149632    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"]
May 20 16:32:35.152555 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:32:35.152527    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681da0df_e4f2_4a57_9863_d9914c41178f.slice/crio-47263cb81e6ca6bc02edd63a4d89d444f861a933ce0ff2c3ae172e80ef7a1719 WatchSource:0}: Error finding container 47263cb81e6ca6bc02edd63a4d89d444f861a933ce0ff2c3ae172e80ef7a1719: Status 404 returned error can't find the container with id 47263cb81e6ca6bc02edd63a4d89d444f861a933ce0ff2c3ae172e80ef7a1719
May 20 16:32:36.140881 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:36.140848    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" event={"ID":"681da0df-e4f2-4a57-9863-d9914c41178f","Type":"ContainerStarted","Data":"1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447"}
May 20 16:32:36.140881 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:36.140883    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" event={"ID":"681da0df-e4f2-4a57-9863-d9914c41178f","Type":"ContainerStarted","Data":"47263cb81e6ca6bc02edd63a4d89d444f861a933ce0ff2c3ae172e80ef7a1719"}
May 20 16:32:37.144969 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:37.144931    2556 generic.go:358] "Generic (PLEG): container finished" podID="681da0df-e4f2-4a57-9863-d9914c41178f" containerID="1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447" exitCode=0
May 20 16:32:37.145359 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:37.144986    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" event={"ID":"681da0df-e4f2-4a57-9863-d9914c41178f","Type":"ContainerDied","Data":"1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447"}
May 20 16:32:38.148887 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.148854    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" event={"ID":"681da0df-e4f2-4a57-9863-d9914c41178f","Type":"ContainerStarted","Data":"aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515"}
May 20 16:32:38.148887 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.148889    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" event={"ID":"681da0df-e4f2-4a57-9863-d9914c41178f","Type":"ContainerStarted","Data":"7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2"}
May 20 16:32:38.149344 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.149056    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:38.149344 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.149183    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:38.150300 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.150274    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
May 20 16:32:38.169786 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.169744    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podStartSLOduration=4.169732483 podStartE2EDuration="4.169732483s" podCreationTimestamp="2026-05-20 16:32:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:32:38.167727516 +0000 UTC m=+3600.858983465" watchObservedRunningTime="2026-05-20 16:32:38.169732483 +0000 UTC m=+3600.860988411"
May 20 16:32:38.691316 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.691290    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:32:38.795877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.795790    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zskr\" (UniqueName: \"kubernetes.io/projected/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kube-api-access-4zskr\") pod \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") "
May 20 16:32:38.795877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.795874    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") "
May 20 16:32:38.796133 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.795909    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kserve-provision-location\") pod \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") "
May 20 16:32:38.796133 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.795934    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-proxy-tls\") pod \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\" (UID: \"542f44d0-1bb3-49f8-8881-6fe1c2fe6805\") "
May 20 16:32:38.796289 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.796259    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "542f44d0-1bb3-49f8-8881-6fe1c2fe6805" (UID: "542f44d0-1bb3-49f8-8881-6fe1c2fe6805"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:32:38.796403 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.796291    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "542f44d0-1bb3-49f8-8881-6fe1c2fe6805" (UID: "542f44d0-1bb3-49f8-8881-6fe1c2fe6805"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:32:38.797960 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.797933    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "542f44d0-1bb3-49f8-8881-6fe1c2fe6805" (UID: "542f44d0-1bb3-49f8-8881-6fe1c2fe6805"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:32:38.798047 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.797957    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kube-api-access-4zskr" (OuterVolumeSpecName: "kube-api-access-4zskr") pod "542f44d0-1bb3-49f8-8881-6fe1c2fe6805" (UID: "542f44d0-1bb3-49f8-8881-6fe1c2fe6805"). InnerVolumeSpecName "kube-api-access-4zskr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:32:38.896886 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.896849    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:32:38.896886 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.896879    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zskr\" (UniqueName: \"kubernetes.io/projected/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kube-api-access-4zskr\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:32:38.897080 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.896894    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:32:38.897080 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:38.896909    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/542f44d0-1bb3-49f8-8881-6fe1c2fe6805-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:32:39.153083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.153002    2556 generic.go:358] "Generic (PLEG): container finished" podID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerID="418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38" exitCode=0
May 20 16:32:39.153464 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.153089    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"
May 20 16:32:39.153464 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.153088    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" event={"ID":"542f44d0-1bb3-49f8-8881-6fe1c2fe6805","Type":"ContainerDied","Data":"418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38"}
May 20 16:32:39.153464 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.153221    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf" event={"ID":"542f44d0-1bb3-49f8-8881-6fe1c2fe6805","Type":"ContainerDied","Data":"2380a58181496579911d1de2dcc0da80859c39da72f07021f311802fe72e0866"}
May 20 16:32:39.153464 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.153244    2556 scope.go:117] "RemoveContainer" containerID="47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2"
May 20 16:32:39.153857 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.153814    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
May 20 16:32:39.160862 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.160846    2556 scope.go:117] "RemoveContainer" containerID="418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38"
May 20 16:32:39.172529 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.172511    2556 scope.go:117] "RemoveContainer" containerID="8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2"
May 20 16:32:39.178641 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.178618    2556 scope.go:117] "RemoveContainer" containerID="47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2"
May 20 16:32:39.178858 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:32:39.178839    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2\": container with ID starting with 47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2 not found: ID does not exist" containerID="47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2"
May 20 16:32:39.178930 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.178869    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2"} err="failed to get container status \"47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2\": rpc error: code = NotFound desc = could not find container \"47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2\": container with ID starting with 47b1ea1dd1a3909b7af5cad64fd49de7a0a08ff804c2891c6aa1320d049e76b2 not found: ID does not exist"
May 20 16:32:39.178930 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.178893    2556 scope.go:117] "RemoveContainer" containerID="418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38"
May 20 16:32:39.179132 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:32:39.179116    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38\": container with ID starting with 418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38 not found: ID does not exist" containerID="418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38"
May 20 16:32:39.179193 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.179137    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38"} err="failed to get container status \"418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38\": rpc error: code = NotFound desc = could not find container \"418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38\": container with ID starting with 418c6050bd6e665d61c247cdf11f436bdf52b75a21ca1aac5c6b5c070b054e38 not found: ID does not exist"
May 20 16:32:39.179193 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.179151    2556 scope.go:117] "RemoveContainer" containerID="8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2"
May 20 16:32:39.179438 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:32:39.179410    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2\": container with ID starting with 8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2 not found: ID does not exist" containerID="8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2"
May 20 16:32:39.179489 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.179446    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2"} err="failed to get container status \"8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2\": rpc error: code = NotFound desc = could not find container \"8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2\": container with ID starting with 8063ae4647296593a5e706aa53b4095c3ff06cdc13b82d09028f4c80c33146d2 not found: ID does not exist"
May 20 16:32:39.181350 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.181329    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"]
May 20 16:32:39.187501 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.187482    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6dd5f9fccb-m4snf"]
May 20 16:32:39.891873 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:39.891842    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" path="/var/lib/kubelet/pods/542f44d0-1bb3-49f8-8881-6fe1c2fe6805/volumes"
May 20 16:32:44.159349 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:44.159314    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:32:44.159840 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:44.159811    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
May 20 16:32:54.160696 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:32:54.160658    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
May 20 16:33:04.159944 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:04.159910    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
May 20 16:33:14.160250 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:14.160207    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
May 20 16:33:24.159904 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:24.159866    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
May 20 16:33:34.160013 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:34.159975    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
May 20 16:33:44.161032 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:44.160997    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:33:44.772047 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:44.772013    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"]
May 20 16:33:44.772332 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:44.772307    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container" containerID="cri-o://7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2" gracePeriod=30
May 20 16:33:44.772378 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:44.772357    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kube-rbac-proxy" containerID="cri-o://aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515" gracePeriod=30
May 20 16:33:45.332257 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.332225    2556 generic.go:358] "Generic (PLEG): container finished" podID="681da0df-e4f2-4a57-9863-d9914c41178f" containerID="aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515" exitCode=2
May 20 16:33:45.332619 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.332305    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" event={"ID":"681da0df-e4f2-4a57-9863-d9914c41178f","Type":"ContainerDied","Data":"aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515"}
May 20 16:33:45.828791 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.828757    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"]
May 20 16:33:45.829038 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.829025    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kube-rbac-proxy"
May 20 16:33:45.829083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.829040    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kube-rbac-proxy"
May 20 16:33:45.829083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.829051    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="storage-initializer"
May 20 16:33:45.829083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.829056    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="storage-initializer"
May 20 16:33:45.829083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.829073    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container"
May 20 16:33:45.829083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.829080    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container"
May 20 16:33:45.829271 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.829157    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kube-rbac-proxy"
May 20 16:33:45.829271 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.829169    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="542f44d0-1bb3-49f8-8881-6fe1c2fe6805" containerName="kserve-container"
May 20 16:33:45.832043 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.832026    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:45.834460 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.834439    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
May 20 16:33:45.834724 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.834707    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
May 20 16:33:45.840013 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.839989    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"]
May 20 16:33:45.954858 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.954831    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:45.954977 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.954869    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4qm\" (UniqueName: \"kubernetes.io/projected/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kube-api-access-4l4qm\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:45.954977 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.954947    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e786424-2d7d-4177-b7ae-ff29ec4c574c-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:45.955122 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:45.955003    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e786424-2d7d-4177-b7ae-ff29ec4c574c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.055638 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.055616    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e786424-2d7d-4177-b7ae-ff29ec4c574c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.055741 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.055654    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.055741 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.055704    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4qm\" (UniqueName: \"kubernetes.io/projected/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kube-api-access-4l4qm\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.055840 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.055774    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e786424-2d7d-4177-b7ae-ff29ec4c574c-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.056044 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.056014    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.056367 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.056345    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e786424-2d7d-4177-b7ae-ff29ec4c574c-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.058259 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.058231    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e786424-2d7d-4177-b7ae-ff29ec4c574c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.064355 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.064330    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4qm\" (UniqueName: \"kubernetes.io/projected/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kube-api-access-4l4qm\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.142607 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.142556    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:46.264947 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.264815    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"]
May 20 16:33:46.267422 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:33:46.267399    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e786424_2d7d_4177_b7ae_ff29ec4c574c.slice/crio-f8d63f8f4e80e4714e1158ad6412a29ae1cc7ffbf703d49b9bdc42b4580104a6 WatchSource:0}: Error finding container f8d63f8f4e80e4714e1158ad6412a29ae1cc7ffbf703d49b9bdc42b4580104a6: Status 404 returned error can't find the container with id f8d63f8f4e80e4714e1158ad6412a29ae1cc7ffbf703d49b9bdc42b4580104a6
May 20 16:33:46.336199 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.336174    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l" event={"ID":"7e786424-2d7d-4177-b7ae-ff29ec4c574c","Type":"ContainerStarted","Data":"0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736"}
May 20 16:33:46.336491 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:46.336206    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l" event={"ID":"7e786424-2d7d-4177-b7ae-ff29ec4c574c","Type":"ContainerStarted","Data":"f8d63f8f4e80e4714e1158ad6412a29ae1cc7ffbf703d49b9bdc42b4580104a6"}
May 20 16:33:48.613597 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.613575    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:33:48.773189 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.773151    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"681da0df-e4f2-4a57-9863-d9914c41178f\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") "
May 20 16:33:48.773354 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.773206    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cq4s\" (UniqueName: \"kubernetes.io/projected/681da0df-e4f2-4a57-9863-d9914c41178f-kube-api-access-6cq4s\") pod \"681da0df-e4f2-4a57-9863-d9914c41178f\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") "
May 20 16:33:48.773354 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.773237    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-cabundle-cert\") pod \"681da0df-e4f2-4a57-9863-d9914c41178f\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") "
May 20 16:33:48.773354 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.773269    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/681da0df-e4f2-4a57-9863-d9914c41178f-kserve-provision-location\") pod \"681da0df-e4f2-4a57-9863-d9914c41178f\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") "
May 20 16:33:48.773354 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.773324    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/681da0df-e4f2-4a57-9863-d9914c41178f-proxy-tls\") pod \"681da0df-e4f2-4a57-9863-d9914c41178f\" (UID: \"681da0df-e4f2-4a57-9863-d9914c41178f\") "
May 20 16:33:48.773678 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.773656    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681da0df-e4f2-4a57-9863-d9914c41178f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "681da0df-e4f2-4a57-9863-d9914c41178f" (UID: "681da0df-e4f2-4a57-9863-d9914c41178f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:33:48.773743 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.773667    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "681da0df-e4f2-4a57-9863-d9914c41178f" (UID: "681da0df-e4f2-4a57-9863-d9914c41178f"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:33:48.773743 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.773686    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "681da0df-e4f2-4a57-9863-d9914c41178f" (UID: "681da0df-e4f2-4a57-9863-d9914c41178f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:33:48.775430 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.775409    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681da0df-e4f2-4a57-9863-d9914c41178f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "681da0df-e4f2-4a57-9863-d9914c41178f" (UID: "681da0df-e4f2-4a57-9863-d9914c41178f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:33:48.775525 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.775486    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681da0df-e4f2-4a57-9863-d9914c41178f-kube-api-access-6cq4s" (OuterVolumeSpecName: "kube-api-access-6cq4s") pod "681da0df-e4f2-4a57-9863-d9914c41178f" (UID: "681da0df-e4f2-4a57-9863-d9914c41178f"). InnerVolumeSpecName "kube-api-access-6cq4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:33:48.874228 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.874197    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cq4s\" (UniqueName: \"kubernetes.io/projected/681da0df-e4f2-4a57-9863-d9914c41178f-kube-api-access-6cq4s\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:48.874228 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.874224    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-cabundle-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:48.874228 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.874235    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/681da0df-e4f2-4a57-9863-d9914c41178f-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:48.874401 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.874245    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/681da0df-e4f2-4a57-9863-d9914c41178f-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:48.874401 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:48.874255    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/681da0df-e4f2-4a57-9863-d9914c41178f-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:49.346322 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.346292    2556 generic.go:358] "Generic (PLEG): container finished" podID="681da0df-e4f2-4a57-9863-d9914c41178f" containerID="7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2" exitCode=0
May 20 16:33:49.346492 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.346332    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" event={"ID":"681da0df-e4f2-4a57-9863-d9914c41178f","Type":"ContainerDied","Data":"7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2"}
May 20 16:33:49.346492 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.346354    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7" event={"ID":"681da0df-e4f2-4a57-9863-d9914c41178f","Type":"ContainerDied","Data":"47263cb81e6ca6bc02edd63a4d89d444f861a933ce0ff2c3ae172e80ef7a1719"}
May 20 16:33:49.346492 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.346370    2556 scope.go:117] "RemoveContainer" containerID="aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515"
May 20 16:33:49.346492 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.346394    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"
May 20 16:33:49.354535 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.354335    2556 scope.go:117] "RemoveContainer" containerID="7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2"
May 20 16:33:49.363508 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.363484    2556 scope.go:117] "RemoveContainer" containerID="1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447"
May 20 16:33:49.368525 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.368496    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"]
May 20 16:33:49.370437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.370414    2556 scope.go:117] "RemoveContainer" containerID="aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515"
May 20 16:33:49.370696 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:33:49.370677    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515\": container with ID starting with aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515 not found: ID does not exist" containerID="aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515"
May 20 16:33:49.370748 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.370706    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515"} err="failed to get container status \"aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515\": rpc error: code = NotFound desc = could not find container \"aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515\": container with ID starting with aaec533de61fd36a7d99a923526d81f3a9d895206b57184d2bc8ef532fc37515 not found: ID does not exist"
May 20 16:33:49.370748 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.370728    2556 scope.go:117] "RemoveContainer" containerID="7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2"
May 20 16:33:49.370963 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:33:49.370946    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2\": container with ID starting with 7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2 not found: ID does not exist" containerID="7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2"
May 20 16:33:49.371034 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.370973    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2"} err="failed to get container status \"7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2\": rpc error: code = NotFound desc = could not find container \"7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2\": container with ID starting with 7e792bf7af80e9fc5bd9aa934e5ebc7441ca89350ecf3ab0e204fb440b6676b2 not found: ID does not exist"
May 20 16:33:49.371034 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.370995    2556 scope.go:117] "RemoveContainer" containerID="1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447"
May 20 16:33:49.371242 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:33:49.371224    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447\": container with ID starting with 1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447 not found: ID does not exist" containerID="1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447"
May 20 16:33:49.371284 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.371248    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447"} err="failed to get container status \"1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447\": rpc error: code = NotFound desc = could not find container \"1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447\": container with ID starting with 1a3d311266d8d9c338878c5812f41811a4c70301df0601e43e2b9bfbbc3d6447 not found: ID does not exist"
May 20 16:33:49.373855 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.373836    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-cd7659964-sn5k7"]
May 20 16:33:49.891386 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:49.891358    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" path="/var/lib/kubelet/pods/681da0df-e4f2-4a57-9863-d9914c41178f/volumes"
May 20 16:33:50.351275 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:50.351252    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l_7e786424-2d7d-4177-b7ae-ff29ec4c574c/storage-initializer/0.log"
May 20 16:33:50.351398 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:50.351288    2556 generic.go:358] "Generic (PLEG): container finished" podID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerID="0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736" exitCode=1
May 20 16:33:50.351398 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:50.351362    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l" event={"ID":"7e786424-2d7d-4177-b7ae-ff29ec4c574c","Type":"ContainerDied","Data":"0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736"}
May 20 16:33:51.356026 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:51.355996    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l_7e786424-2d7d-4177-b7ae-ff29ec4c574c/storage-initializer/0.log"
May 20 16:33:51.356465 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:51.356058    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l" event={"ID":"7e786424-2d7d-4177-b7ae-ff29ec4c574c","Type":"ContainerStarted","Data":"10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6"}
May 20 16:33:55.848021 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:55.847994    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"]
May 20 16:33:55.848372 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:55.848304    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l" podUID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerName="storage-initializer" containerID="cri-o://10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6" gracePeriod=30
May 20 16:33:55.980878 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:55.980855    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l_7e786424-2d7d-4177-b7ae-ff29ec4c574c/storage-initializer/1.log"
May 20 16:33:55.981223 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:55.981207    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l_7e786424-2d7d-4177-b7ae-ff29ec4c574c/storage-initializer/0.log"
May 20 16:33:55.981316 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:55.981283    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:56.025566 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.025508    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kserve-provision-location\") pod \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") "
May 20 16:33:56.025566 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.025545    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e786424-2d7d-4177-b7ae-ff29ec4c574c-proxy-tls\") pod \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") "
May 20 16:33:56.025696 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.025594    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l4qm\" (UniqueName: \"kubernetes.io/projected/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kube-api-access-4l4qm\") pod \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") "
May 20 16:33:56.025788 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.025770    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e786424-2d7d-4177-b7ae-ff29ec4c574c" (UID: "7e786424-2d7d-4177-b7ae-ff29ec4c574c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:33:56.027527 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.027504    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e786424-2d7d-4177-b7ae-ff29ec4c574c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e786424-2d7d-4177-b7ae-ff29ec4c574c" (UID: "7e786424-2d7d-4177-b7ae-ff29ec4c574c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:33:56.027614 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.027568    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kube-api-access-4l4qm" (OuterVolumeSpecName: "kube-api-access-4l4qm") pod "7e786424-2d7d-4177-b7ae-ff29ec4c574c" (UID: "7e786424-2d7d-4177-b7ae-ff29ec4c574c"). InnerVolumeSpecName "kube-api-access-4l4qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:33:56.126753 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.126718    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e786424-2d7d-4177-b7ae-ff29ec4c574c-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\" (UID: \"7e786424-2d7d-4177-b7ae-ff29ec4c574c\") "
May 20 16:33:56.126927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.126854    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:56.126927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.126865    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e786424-2d7d-4177-b7ae-ff29ec4c574c-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:56.126927 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.126874    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l4qm\" (UniqueName: \"kubernetes.io/projected/7e786424-2d7d-4177-b7ae-ff29ec4c574c-kube-api-access-4l4qm\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:56.127064 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.127041    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e786424-2d7d-4177-b7ae-ff29ec4c574c-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "7e786424-2d7d-4177-b7ae-ff29ec4c574c" (UID: "7e786424-2d7d-4177-b7ae-ff29ec4c574c"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:33:56.227418 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.227356    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e786424-2d7d-4177-b7ae-ff29ec4c574c-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:33:56.371808 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.371742    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l_7e786424-2d7d-4177-b7ae-ff29ec4c574c/storage-initializer/1.log"
May 20 16:33:56.372149 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.372129    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l_7e786424-2d7d-4177-b7ae-ff29ec4c574c/storage-initializer/0.log"
May 20 16:33:56.372227 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.372166    2556 generic.go:358] "Generic (PLEG): container finished" podID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerID="10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6" exitCode=1
May 20 16:33:56.372227 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.372212    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l" event={"ID":"7e786424-2d7d-4177-b7ae-ff29ec4c574c","Type":"ContainerDied","Data":"10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6"}
May 20 16:33:56.372296 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.372233    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l" event={"ID":"7e786424-2d7d-4177-b7ae-ff29ec4c574c","Type":"ContainerDied","Data":"f8d63f8f4e80e4714e1158ad6412a29ae1cc7ffbf703d49b9bdc42b4580104a6"}
May 20 16:33:56.372296 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.372248    2556 scope.go:117] "RemoveContainer" containerID="10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6"
May 20 16:33:56.372296 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.372252    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"
May 20 16:33:56.379957 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.379940    2556 scope.go:117] "RemoveContainer" containerID="0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736"
May 20 16:33:56.386617 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.386599    2556 scope.go:117] "RemoveContainer" containerID="10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6"
May 20 16:33:56.386848 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:33:56.386830    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6\": container with ID starting with 10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6 not found: ID does not exist" containerID="10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6"
May 20 16:33:56.386906 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.386856    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6"} err="failed to get container status \"10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6\": rpc error: code = NotFound desc = could not find container \"10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6\": container with ID starting with 10aa5219e9eed5508460b927d9b1a45c65cf0f144b010fcbe0af494ea5dbe6e6 not found: ID does not exist"
May 20 16:33:56.386906 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.386873    2556 scope.go:117] "RemoveContainer" containerID="0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736"
May 20 16:33:56.387129 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:33:56.387112    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736\": container with ID starting with 0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736 not found: ID does not exist" containerID="0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736"
May 20 16:33:56.387171 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.387135    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736"} err="failed to get container status \"0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736\": rpc error: code = NotFound desc = could not find container \"0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736\": container with ID starting with 0e3bec65326c6ecb56e40d8cbc0744c7bbf3ab12afc66d7b4e7a5e8ea03b2736 not found: ID does not exist"
May 20 16:33:56.408969 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.408947    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"]
May 20 16:33:56.413196 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.413176    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-bb46758c8-j5r5l"]
May 20 16:33:56.918073 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918043    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"]
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918292    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918303    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918317    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerName="storage-initializer"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918323    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerName="storage-initializer"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918333    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kube-rbac-proxy"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918339    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kube-rbac-proxy"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918347    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="storage-initializer"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918353    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="storage-initializer"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918362    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerName="storage-initializer"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918367    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerName="storage-initializer"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918406    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerName="storage-initializer"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918415    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kserve-container"
May 20 16:33:56.918435 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918422    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="681da0df-e4f2-4a57-9863-d9914c41178f" containerName="kube-rbac-proxy"
May 20 16:33:56.919059 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.918497    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" containerName="storage-initializer"
May 20 16:33:56.922675 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.922658    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:56.925458 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.925432    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
May 20 16:33:56.925582 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.925478    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 20 16:33:56.925582 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.925492    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
May 20 16:33:56.925582 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.925480    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fxjhf\""
May 20 16:33:56.925582 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.925575    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 16:33:56.926492 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.926476    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 16:33:56.926611 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.926538    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 16:33:56.929914 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.929892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:56.930014 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.929944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:56.930014 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.929974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5j67\" (UniqueName: \"kubernetes.io/projected/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kube-api-access-f5j67\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:56.930146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.930027    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:56.930146 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.930093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:56.931993 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:56.931969    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"]
May 20 16:33:57.030985 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.030956    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.031084 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.030998    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.031084 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.031028    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.031084 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.031055    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f5j67\" (UniqueName: \"kubernetes.io/projected/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kube-api-access-f5j67\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.031266 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.031086    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.031266 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:33:57.031222    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
May 20 16:33:57.031373 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:33:57.031284    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls podName:054e85f1-ec3b-410c-b36b-c6faaf0f4af2 nodeName:}" failed. No retries permitted until 2026-05-20 16:33:57.531268203 +0000 UTC m=+3680.222524109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" (UID: "054e85f1-ec3b-410c-b36b-c6faaf0f4af2") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
May 20 16:33:57.031437 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.031396    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.031712 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.031689    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.031797 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.031748    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.042929 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.042901    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5j67\" (UniqueName: \"kubernetes.io/projected/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kube-api-access-f5j67\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.534881 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.534846    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.537247 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.537228    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.834200 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.834121    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:33:57.892949 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.892920    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e786424-2d7d-4177-b7ae-ff29ec4c574c" path="/var/lib/kubelet/pods/7e786424-2d7d-4177-b7ae-ff29ec4c574c/volumes"
May 20 16:33:57.958972 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:57.958946    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"]
May 20 16:33:57.961833 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:33:57.961809    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054e85f1_ec3b_410c_b36b_c6faaf0f4af2.slice/crio-d603ae8b6d47f17ce10d10c3d7197248281743f985b9e12f5917b14e4aa864a6 WatchSource:0}: Error finding container d603ae8b6d47f17ce10d10c3d7197248281743f985b9e12f5917b14e4aa864a6: Status 404 returned error can't find the container with id d603ae8b6d47f17ce10d10c3d7197248281743f985b9e12f5917b14e4aa864a6
May 20 16:33:58.380777 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:58.380734    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" event={"ID":"054e85f1-ec3b-410c-b36b-c6faaf0f4af2","Type":"ContainerStarted","Data":"7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0"}
May 20 16:33:58.380777 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:58.380777    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" event={"ID":"054e85f1-ec3b-410c-b36b-c6faaf0f4af2","Type":"ContainerStarted","Data":"d603ae8b6d47f17ce10d10c3d7197248281743f985b9e12f5917b14e4aa864a6"}
May 20 16:33:59.385853 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:59.385762    2556 generic.go:358] "Generic (PLEG): container finished" podID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerID="7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0" exitCode=0
May 20 16:33:59.386306 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:33:59.385858    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" event={"ID":"054e85f1-ec3b-410c-b36b-c6faaf0f4af2","Type":"ContainerDied","Data":"7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0"}
May 20 16:34:00.396083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:00.396044    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" event={"ID":"054e85f1-ec3b-410c-b36b-c6faaf0f4af2","Type":"ContainerStarted","Data":"163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1"}
May 20 16:34:00.396083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:00.396080    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" event={"ID":"054e85f1-ec3b-410c-b36b-c6faaf0f4af2","Type":"ContainerStarted","Data":"b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17"}
May 20 16:34:00.396662 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:00.396197    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:34:00.416399 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:00.416357    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podStartSLOduration=4.416344423 podStartE2EDuration="4.416344423s" podCreationTimestamp="2026-05-20 16:33:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:34:00.415286615 +0000 UTC m=+3683.106542545" watchObservedRunningTime="2026-05-20 16:34:00.416344423 +0000 UTC m=+3683.107600350"
May 20 16:34:01.399026 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:01.398990    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:34:01.400235 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:01.400206    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:34:02.401159 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:02.401119    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:34:07.410303 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:07.410274    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:34:07.410835 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:07.410810    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:34:17.411736 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:17.411683    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:34:27.411248 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:27.411202    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:34:37.411348 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:37.411309    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:34:47.411372 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:47.411323    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:34:57.411717 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:34:57.411673    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:35:07.411628 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:07.411599    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:35:16.966192 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:16.966161    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"]
May 20 16:35:16.966589 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:16.966475    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" containerID="cri-o://b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17" gracePeriod=30
May 20 16:35:16.966589 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:16.966532    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kube-rbac-proxy" containerID="cri-o://163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1" gracePeriod=30
May 20 16:35:17.401934 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:17.401893    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
May 20 16:35:17.411729 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:17.411699    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
May 20 16:35:17.600135 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:17.600093    2556 generic.go:358] "Generic (PLEG): container finished" podID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerID="163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1" exitCode=2
May 20 16:35:17.600271 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:17.600132    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" event={"ID":"054e85f1-ec3b-410c-b36b-c6faaf0f4af2","Type":"ContainerDied","Data":"163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1"}
May 20 16:35:18.034093 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.034055    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"]
May 20 16:35:18.037321 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.037298    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.039621 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.039598    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
May 20 16:35:18.039722 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.039648    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
May 20 16:35:18.046083 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.046062    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"]
May 20 16:35:18.207869 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.207826    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d0e7108-4a7a-403f-8c23-a25358fd6531-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.207869 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.207875    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d0e7108-4a7a-403f-8c23-a25358fd6531-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.208193 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.207962    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdpnk\" (UniqueName: \"kubernetes.io/projected/3d0e7108-4a7a-403f-8c23-a25358fd6531-kube-api-access-sdpnk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.208193 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.207996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d0e7108-4a7a-403f-8c23-a25358fd6531-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.309297 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.309202    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpnk\" (UniqueName: \"kubernetes.io/projected/3d0e7108-4a7a-403f-8c23-a25358fd6531-kube-api-access-sdpnk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.309297 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.309240    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d0e7108-4a7a-403f-8c23-a25358fd6531-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.309297 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.309286    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d0e7108-4a7a-403f-8c23-a25358fd6531-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.309579 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.309314    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d0e7108-4a7a-403f-8c23-a25358fd6531-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.309725 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.309702    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d0e7108-4a7a-403f-8c23-a25358fd6531-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.309974 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.309953    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d0e7108-4a7a-403f-8c23-a25358fd6531-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.311842 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.311815    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d0e7108-4a7a-403f-8c23-a25358fd6531-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.318358 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.318338    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpnk\" (UniqueName: \"kubernetes.io/projected/3d0e7108-4a7a-403f-8c23-a25358fd6531-kube-api-access-sdpnk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.347307 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.347280    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:18.474691 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.474658    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"]
May 20 16:35:18.477743 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:35:18.477721    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0e7108_4a7a_403f_8c23_a25358fd6531.slice/crio-1c2f9e533a1db0e08a88dc20108433549d5ff6e1506e34d16b221350cbc9817a WatchSource:0}: Error finding container 1c2f9e533a1db0e08a88dc20108433549d5ff6e1506e34d16b221350cbc9817a: Status 404 returned error can't find the container with id 1c2f9e533a1db0e08a88dc20108433549d5ff6e1506e34d16b221350cbc9817a
May 20 16:35:18.604733 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.604639    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz" event={"ID":"3d0e7108-4a7a-403f-8c23-a25358fd6531","Type":"ContainerStarted","Data":"0d5594d2103c1a74e2354f4a9ac7f57a2758a1449cf3a8e016a06b84974f5e5c"}
May 20 16:35:18.604733 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:18.604679    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz" event={"ID":"3d0e7108-4a7a-403f-8c23-a25358fd6531","Type":"ContainerStarted","Data":"1c2f9e533a1db0e08a88dc20108433549d5ff6e1506e34d16b221350cbc9817a"}
May 20 16:35:21.105576 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.105555    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:35:21.232350 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.232272    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-cabundle-cert\") pod \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") "
May 20 16:35:21.232350 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.232318    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") "
May 20 16:35:21.232578 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.232353    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kserve-provision-location\") pod \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") "
May 20 16:35:21.232578 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.232391    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5j67\" (UniqueName: \"kubernetes.io/projected/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kube-api-access-f5j67\") pod \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") "
May 20 16:35:21.232578 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.232422    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls\") pod \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\" (UID: \"054e85f1-ec3b-410c-b36b-c6faaf0f4af2\") "
May 20 16:35:21.232755 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.232658    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "054e85f1-ec3b-410c-b36b-c6faaf0f4af2" (UID: "054e85f1-ec3b-410c-b36b-c6faaf0f4af2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:35:21.232755 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.232730    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "054e85f1-ec3b-410c-b36b-c6faaf0f4af2" (UID: "054e85f1-ec3b-410c-b36b-c6faaf0f4af2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:35:21.232755 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.232748    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "054e85f1-ec3b-410c-b36b-c6faaf0f4af2" (UID: "054e85f1-ec3b-410c-b36b-c6faaf0f4af2"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:35:21.234505 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.234482    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "054e85f1-ec3b-410c-b36b-c6faaf0f4af2" (UID: "054e85f1-ec3b-410c-b36b-c6faaf0f4af2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:35:21.234630 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.234610    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kube-api-access-f5j67" (OuterVolumeSpecName: "kube-api-access-f5j67") pod "054e85f1-ec3b-410c-b36b-c6faaf0f4af2" (UID: "054e85f1-ec3b-410c-b36b-c6faaf0f4af2"). InnerVolumeSpecName "kube-api-access-f5j67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:35:21.333069 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.333045    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:21.333069 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.333065    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:21.333248 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.333075    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5j67\" (UniqueName: \"kubernetes.io/projected/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-kube-api-access-f5j67\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:21.333248 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.333086    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:21.333248 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.333113    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054e85f1-ec3b-410c-b36b-c6faaf0f4af2-cabundle-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:21.613768 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.613731    2556 generic.go:358] "Generic (PLEG): container finished" podID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerID="b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17" exitCode=0
May 20 16:35:21.613916 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.613814    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"
May 20 16:35:21.613916 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.613814    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" event={"ID":"054e85f1-ec3b-410c-b36b-c6faaf0f4af2","Type":"ContainerDied","Data":"b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17"}
May 20 16:35:21.613916 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.613857    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5" event={"ID":"054e85f1-ec3b-410c-b36b-c6faaf0f4af2","Type":"ContainerDied","Data":"d603ae8b6d47f17ce10d10c3d7197248281743f985b9e12f5917b14e4aa864a6"}
May 20 16:35:21.613916 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.613877    2556 scope.go:117] "RemoveContainer" containerID="163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1"
May 20 16:35:21.629067 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.629047    2556 scope.go:117] "RemoveContainer" containerID="b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17"
May 20 16:35:21.635523 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.635506    2556 scope.go:117] "RemoveContainer" containerID="7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0"
May 20 16:35:21.640318 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.640294    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"]
May 20 16:35:21.642379 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.642358    2556 scope.go:117] "RemoveContainer" containerID="163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1"
May 20 16:35:21.642633 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:35:21.642613    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1\": container with ID starting with 163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1 not found: ID does not exist" containerID="163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1"
May 20 16:35:21.642689 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.642646    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1"} err="failed to get container status \"163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1\": rpc error: code = NotFound desc = could not find container \"163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1\": container with ID starting with 163960b9f6dd485dc0d94676711495eab3e939f0c1d697af669393c1807608d1 not found: ID does not exist"
May 20 16:35:21.642689 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.642668    2556 scope.go:117] "RemoveContainer" containerID="b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17"
May 20 16:35:21.642908 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:35:21.642891    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17\": container with ID starting with b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17 not found: ID does not exist" containerID="b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17"
May 20 16:35:21.642952 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.642914    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17"} err="failed to get container status \"b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17\": rpc error: code = NotFound desc = could not find container \"b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17\": container with ID starting with b68ce1acceb03bab494aff10b65abce7c5e4cfc4dde1e6dd5397beb6083caa17 not found: ID does not exist"
May 20 16:35:21.642952 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.642930    2556 scope.go:117] "RemoveContainer" containerID="7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0"
May 20 16:35:21.643182 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:35:21.643166    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0\": container with ID starting with 7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0 not found: ID does not exist" containerID="7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0"
May 20 16:35:21.643222 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.643195    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0"} err="failed to get container status \"7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0\": rpc error: code = NotFound desc = could not find container \"7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0\": container with ID starting with 7015622d4f56b4b74aa57d4f6b5458e885a94781aaf9bb4e45e20253cb7db6a0 not found: ID does not exist"
May 20 16:35:21.645796 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.645770    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76798fc596-jbkh5"]
May 20 16:35:21.892465 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:21.892390    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" path="/var/lib/kubelet/pods/054e85f1-ec3b-410c-b36b-c6faaf0f4af2/volumes"
May 20 16:35:23.621086 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:23.621056    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_3d0e7108-4a7a-403f-8c23-a25358fd6531/storage-initializer/0.log"
May 20 16:35:23.621493 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:23.621115    2556 generic.go:358] "Generic (PLEG): container finished" podID="3d0e7108-4a7a-403f-8c23-a25358fd6531" containerID="0d5594d2103c1a74e2354f4a9ac7f57a2758a1449cf3a8e016a06b84974f5e5c" exitCode=1
May 20 16:35:23.621493 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:23.621186    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz" event={"ID":"3d0e7108-4a7a-403f-8c23-a25358fd6531","Type":"ContainerDied","Data":"0d5594d2103c1a74e2354f4a9ac7f57a2758a1449cf3a8e016a06b84974f5e5c"}
May 20 16:35:24.626115 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:24.626074    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_3d0e7108-4a7a-403f-8c23-a25358fd6531/storage-initializer/0.log"
May 20 16:35:24.626491 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:24.626139    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz" event={"ID":"3d0e7108-4a7a-403f-8c23-a25358fd6531","Type":"ContainerStarted","Data":"337fe53daa7a86992a10717f4d197755e7cb51d9b2e8cf0d4accf5b50cc336a6"}
May 20 16:35:27.637182 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:27.637150    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_3d0e7108-4a7a-403f-8c23-a25358fd6531/storage-initializer/1.log"
May 20 16:35:27.637603 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:27.637532    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_3d0e7108-4a7a-403f-8c23-a25358fd6531/storage-initializer/0.log"
May 20 16:35:27.637603 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:27.637566    2556 generic.go:358] "Generic (PLEG): container finished" podID="3d0e7108-4a7a-403f-8c23-a25358fd6531" containerID="337fe53daa7a86992a10717f4d197755e7cb51d9b2e8cf0d4accf5b50cc336a6" exitCode=1
May 20 16:35:27.637701 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:27.637630    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz" event={"ID":"3d0e7108-4a7a-403f-8c23-a25358fd6531","Type":"ContainerDied","Data":"337fe53daa7a86992a10717f4d197755e7cb51d9b2e8cf0d4accf5b50cc336a6"}
May 20 16:35:27.637701 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:27.637665    2556 scope.go:117] "RemoveContainer" containerID="0d5594d2103c1a74e2354f4a9ac7f57a2758a1449cf3a8e016a06b84974f5e5c"
May 20 16:35:27.638056 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:27.638033    2556 scope.go:117] "RemoveContainer" containerID="0d5594d2103c1a74e2354f4a9ac7f57a2758a1449cf3a8e016a06b84974f5e5c"
May 20 16:35:27.648726 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:35:27.648685    2556 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_kserve-ci-e2e-test_3d0e7108-4a7a-403f-8c23-a25358fd6531_0 in pod sandbox 1c2f9e533a1db0e08a88dc20108433549d5ff6e1506e34d16b221350cbc9817a from index: no such id: '0d5594d2103c1a74e2354f4a9ac7f57a2758a1449cf3a8e016a06b84974f5e5c'" containerID="0d5594d2103c1a74e2354f4a9ac7f57a2758a1449cf3a8e016a06b84974f5e5c"
May 20 16:35:27.648858 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:35:27.648746    2556 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_kserve-ci-e2e-test_3d0e7108-4a7a-403f-8c23-a25358fd6531_0 in pod sandbox 1c2f9e533a1db0e08a88dc20108433549d5ff6e1506e34d16b221350cbc9817a from index: no such id: '0d5594d2103c1a74e2354f4a9ac7f57a2758a1449cf3a8e016a06b84974f5e5c'; Skipping pod \"isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_kserve-ci-e2e-test(3d0e7108-4a7a-403f-8c23-a25358fd6531)\"" logger="UnhandledError"
May 20 16:35:27.650056 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:35:27.650032    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_kserve-ci-e2e-test(3d0e7108-4a7a-403f-8c23-a25358fd6531)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz" podUID="3d0e7108-4a7a-403f-8c23-a25358fd6531"
May 20 16:35:28.045077 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.045046    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"]
May 20 16:35:28.642213 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.642186    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_3d0e7108-4a7a-403f-8c23-a25358fd6531/storage-initializer/1.log"
May 20 16:35:28.761704 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.761682    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_3d0e7108-4a7a-403f-8c23-a25358fd6531/storage-initializer/1.log"
May 20 16:35:28.761808 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.761742    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:28.887900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.887875    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d0e7108-4a7a-403f-8c23-a25358fd6531-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"3d0e7108-4a7a-403f-8c23-a25358fd6531\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") "
May 20 16:35:28.888014 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.887964    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d0e7108-4a7a-403f-8c23-a25358fd6531-proxy-tls\") pod \"3d0e7108-4a7a-403f-8c23-a25358fd6531\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") "
May 20 16:35:28.888014 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.888002    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdpnk\" (UniqueName: \"kubernetes.io/projected/3d0e7108-4a7a-403f-8c23-a25358fd6531-kube-api-access-sdpnk\") pod \"3d0e7108-4a7a-403f-8c23-a25358fd6531\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") "
May 20 16:35:28.888135 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.888058    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d0e7108-4a7a-403f-8c23-a25358fd6531-kserve-provision-location\") pod \"3d0e7108-4a7a-403f-8c23-a25358fd6531\" (UID: \"3d0e7108-4a7a-403f-8c23-a25358fd6531\") "
May 20 16:35:28.888396 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.888375    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0e7108-4a7a-403f-8c23-a25358fd6531-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d0e7108-4a7a-403f-8c23-a25358fd6531" (UID: "3d0e7108-4a7a-403f-8c23-a25358fd6531"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:35:28.889191 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.889162    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0e7108-4a7a-403f-8c23-a25358fd6531-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "3d0e7108-4a7a-403f-8c23-a25358fd6531" (UID: "3d0e7108-4a7a-403f-8c23-a25358fd6531"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:35:28.889937 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.889916    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0e7108-4a7a-403f-8c23-a25358fd6531-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d0e7108-4a7a-403f-8c23-a25358fd6531" (UID: "3d0e7108-4a7a-403f-8c23-a25358fd6531"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:35:28.890139 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.890118    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0e7108-4a7a-403f-8c23-a25358fd6531-kube-api-access-sdpnk" (OuterVolumeSpecName: "kube-api-access-sdpnk") pod "3d0e7108-4a7a-403f-8c23-a25358fd6531" (UID: "3d0e7108-4a7a-403f-8c23-a25358fd6531"). InnerVolumeSpecName "kube-api-access-sdpnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:35:28.989254 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.989230    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdpnk\" (UniqueName: \"kubernetes.io/projected/3d0e7108-4a7a-403f-8c23-a25358fd6531-kube-api-access-sdpnk\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:28.989254 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.989253    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d0e7108-4a7a-403f-8c23-a25358fd6531-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:28.989403 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.989264    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d0e7108-4a7a-403f-8c23-a25358fd6531-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:28.989403 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:28.989273    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d0e7108-4a7a-403f-8c23-a25358fd6531-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:35:29.100392 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100366    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"]
May 20 16:35:29.100595 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100583    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="storage-initializer"
May 20 16:35:29.100647 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100597    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="storage-initializer"
May 20 16:35:29.100647 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100609    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0e7108-4a7a-403f-8c23-a25358fd6531" containerName="storage-initializer"
May 20 16:35:29.100647 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100614    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0e7108-4a7a-403f-8c23-a25358fd6531" containerName="storage-initializer"
May 20 16:35:29.100647 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100621    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0e7108-4a7a-403f-8c23-a25358fd6531" containerName="storage-initializer"
May 20 16:35:29.100647 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100627    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0e7108-4a7a-403f-8c23-a25358fd6531" containerName="storage-initializer"
May 20 16:35:29.100647 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100636    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kube-rbac-proxy"
May 20 16:35:29.100647 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100642    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kube-rbac-proxy"
May 20 16:35:29.100852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100655    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container"
May 20 16:35:29.100852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100661    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container"
May 20 16:35:29.100852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100699    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kserve-container"
May 20 16:35:29.100852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100707    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="054e85f1-ec3b-410c-b36b-c6faaf0f4af2" containerName="kube-rbac-proxy"
May 20 16:35:29.100852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100714    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d0e7108-4a7a-403f-8c23-a25358fd6531" containerName="storage-initializer"
May 20 16:35:29.100852 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.100721    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d0e7108-4a7a-403f-8c23-a25358fd6531" containerName="storage-initializer"
May 20 16:35:29.104823 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.104807    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.107190 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.107170    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 16:35:29.107379 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.107355    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
May 20 16:35:29.107379 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.107375    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
May 20 16:35:29.114728 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.114709    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"]
May 20 16:35:29.292050 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.292018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b353fb0-a562-4d67-8377-b682f28a1e81-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.292221 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.292059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js46b\" (UniqueName: \"kubernetes.io/projected/4b353fb0-a562-4d67-8377-b682f28a1e81-kube-api-access-js46b\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.292221 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.292089    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.292221 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.292164    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.292221 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.292209    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b353fb0-a562-4d67-8377-b682f28a1e81-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.392759 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.392730    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b353fb0-a562-4d67-8377-b682f28a1e81-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.392877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.392763    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-js46b\" (UniqueName: \"kubernetes.io/projected/4b353fb0-a562-4d67-8377-b682f28a1e81-kube-api-access-js46b\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.392877 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.392790    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.392956 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.392913    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.392998 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.392963    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b353fb0-a562-4d67-8377-b682f28a1e81-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.393383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.393361    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b353fb0-a562-4d67-8377-b682f28a1e81-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.393516 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.393497    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.393582 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.393564    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.395266 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.395248    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b353fb0-a562-4d67-8377-b682f28a1e81-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.401372 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.401352    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js46b\" (UniqueName: \"kubernetes.io/projected/4b353fb0-a562-4d67-8377-b682f28a1e81-kube-api-access-js46b\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.415114 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.415077    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:29.537180 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.537151    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"]
May 20 16:35:29.540249 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:35:29.540226    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b353fb0_a562_4d67_8377_b682f28a1e81.slice/crio-3bad9ca32e59983728c40fbaeb912fa31a190311ec5c0ce8efd457799a2b668f WatchSource:0}: Error finding container 3bad9ca32e59983728c40fbaeb912fa31a190311ec5c0ce8efd457799a2b668f: Status 404 returned error can't find the container with id 3bad9ca32e59983728c40fbaeb912fa31a190311ec5c0ce8efd457799a2b668f
May 20 16:35:29.646121 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.646087    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz_3d0e7108-4a7a-403f-8c23-a25358fd6531/storage-initializer/1.log"
May 20 16:35:29.646459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.646207    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz" event={"ID":"3d0e7108-4a7a-403f-8c23-a25358fd6531","Type":"ContainerDied","Data":"1c2f9e533a1db0e08a88dc20108433549d5ff6e1506e34d16b221350cbc9817a"}
May 20 16:35:29.646459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.646239    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"
May 20 16:35:29.646459 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.646246    2556 scope.go:117] "RemoveContainer" containerID="337fe53daa7a86992a10717f4d197755e7cb51d9b2e8cf0d4accf5b50cc336a6"
May 20 16:35:29.647668 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.647637    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" event={"ID":"4b353fb0-a562-4d67-8377-b682f28a1e81","Type":"ContainerStarted","Data":"0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34"}
May 20 16:35:29.647668 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.647668    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" event={"ID":"4b353fb0-a562-4d67-8377-b682f28a1e81","Type":"ContainerStarted","Data":"3bad9ca32e59983728c40fbaeb912fa31a190311ec5c0ce8efd457799a2b668f"}
May 20 16:35:29.694494 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.694468    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"]
May 20 16:35:29.698506 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.698486    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-84ccbb686b-zc5nz"]
May 20 16:35:29.895250 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:29.892228    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0e7108-4a7a-403f-8c23-a25358fd6531" path="/var/lib/kubelet/pods/3d0e7108-4a7a-403f-8c23-a25358fd6531/volumes"
May 20 16:35:30.654383 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:30.654351    2556 generic.go:358] "Generic (PLEG): container finished" podID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerID="0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34" exitCode=0
May 20 16:35:30.654761 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:30.654393    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" event={"ID":"4b353fb0-a562-4d67-8377-b682f28a1e81","Type":"ContainerDied","Data":"0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34"}
May 20 16:35:31.660718 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:31.660684    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" event={"ID":"4b353fb0-a562-4d67-8377-b682f28a1e81","Type":"ContainerStarted","Data":"16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a"}
May 20 16:35:31.660718 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:31.660719    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" event={"ID":"4b353fb0-a562-4d67-8377-b682f28a1e81","Type":"ContainerStarted","Data":"6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb"}
May 20 16:35:31.661225 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:31.660889    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:31.683569 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:31.683505    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podStartSLOduration=2.6834899549999998 podStartE2EDuration="2.683489955s" podCreationTimestamp="2026-05-20 16:35:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:35:31.682215822 +0000 UTC m=+3774.373471751" watchObservedRunningTime="2026-05-20 16:35:31.683489955 +0000 UTC m=+3774.374745883"
May 20 16:35:32.663633 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:32.663598    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:32.664862 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:32.664833    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
May 20 16:35:33.666456 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:33.666416    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
May 20 16:35:38.670789 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:38.670761    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:35:38.671415 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:38.671387    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
May 20 16:35:48.671599 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:48.671564    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
May 20 16:35:58.672255 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:35:58.672217    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
May 20 16:36:08.671646 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:08.671606    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
May 20 16:36:18.672244 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:18.672206    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
May 20 16:36:28.671869 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:28.671832    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
May 20 16:36:38.671986 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:38.671948    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:36:39.141045 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:39.141007    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"]
May 20 16:36:39.141638 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:39.141397    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container" containerID="cri-o://6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb" gracePeriod=30
May 20 16:36:39.141638 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:39.141478    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kube-rbac-proxy" containerID="cri-o://16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a" gracePeriod=30
May 20 16:36:39.860193 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:39.860162    2556 generic.go:358] "Generic (PLEG): container finished" podID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerID="16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a" exitCode=2
May 20 16:36:39.860551 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:39.860200    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" event={"ID":"4b353fb0-a562-4d67-8377-b682f28a1e81","Type":"ContainerDied","Data":"16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a"}
May 20 16:36:40.333471 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.333428    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"]
May 20 16:36:40.336634 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.336617    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.339509 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.339480    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
May 20 16:36:40.339509 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.339500    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
May 20 16:36:40.347473 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.347452    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"]
May 20 16:36:40.378115 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.378077    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a67aed04-469c-4eb5-855b-68b9196d6210-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.378237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.378123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/a67aed04-469c-4eb5-855b-68b9196d6210-kube-api-access-2kmhw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.378237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.378153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a67aed04-469c-4eb5-855b-68b9196d6210-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.378324 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.378250    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a67aed04-469c-4eb5-855b-68b9196d6210-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.479555 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.479530    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a67aed04-469c-4eb5-855b-68b9196d6210-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.479674 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.479575    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a67aed04-469c-4eb5-855b-68b9196d6210-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.479674 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.479595    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/a67aed04-469c-4eb5-855b-68b9196d6210-kube-api-access-2kmhw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.479800 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.479700    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a67aed04-469c-4eb5-855b-68b9196d6210-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.480081 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.480061    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a67aed04-469c-4eb5-855b-68b9196d6210-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.480186 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.480165    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a67aed04-469c-4eb5-855b-68b9196d6210-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.482232 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.482214    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a67aed04-469c-4eb5-855b-68b9196d6210-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.487919 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.487896    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/a67aed04-469c-4eb5-855b-68b9196d6210-kube-api-access-2kmhw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.646980 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.646915    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:40.767935 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.767907    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"]
May 20 16:36:40.770792 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:36:40.770755    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67aed04_469c_4eb5_855b_68b9196d6210.slice/crio-521dd0af690cb02efca43e532f34c480513f998ac8b00565be577b8624733e99 WatchSource:0}: Error finding container 521dd0af690cb02efca43e532f34c480513f998ac8b00565be577b8624733e99: Status 404 returned error can't find the container with id 521dd0af690cb02efca43e532f34c480513f998ac8b00565be577b8624733e99
May 20 16:36:40.772675 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.772656    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:36:40.869349 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.869313    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn" event={"ID":"a67aed04-469c-4eb5-855b-68b9196d6210","Type":"ContainerStarted","Data":"7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef"}
May 20 16:36:40.869674 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:40.869357    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn" event={"ID":"a67aed04-469c-4eb5-855b-68b9196d6210","Type":"ContainerStarted","Data":"521dd0af690cb02efca43e532f34c480513f998ac8b00565be577b8624733e99"}
May 20 16:36:42.770919 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.770894    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:36:42.875803 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.875740    2556 generic.go:358] "Generic (PLEG): container finished" podID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerID="6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb" exitCode=0
May 20 16:36:42.875803 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.875794    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" event={"ID":"4b353fb0-a562-4d67-8377-b682f28a1e81","Type":"ContainerDied","Data":"6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb"}
May 20 16:36:42.875999 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.875818    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs" event={"ID":"4b353fb0-a562-4d67-8377-b682f28a1e81","Type":"ContainerDied","Data":"3bad9ca32e59983728c40fbaeb912fa31a190311ec5c0ce8efd457799a2b668f"}
May 20 16:36:42.875999 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.875824    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"
May 20 16:36:42.875999 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.875834    2556 scope.go:117] "RemoveContainer" containerID="16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a"
May 20 16:36:42.882736 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.882720    2556 scope.go:117] "RemoveContainer" containerID="6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb"
May 20 16:36:42.889337 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.889297    2556 scope.go:117] "RemoveContainer" containerID="0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34"
May 20 16:36:42.895499 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.895484    2556 scope.go:117] "RemoveContainer" containerID="16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a"
May 20 16:36:42.895736 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:36:42.895719    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a\": container with ID starting with 16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a not found: ID does not exist" containerID="16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a"
May 20 16:36:42.895783 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.895744    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a"} err="failed to get container status \"16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a\": rpc error: code = NotFound desc = could not find container \"16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a\": container with ID starting with 16435d20404fb1034bec4efbf9ef6bd2de6eef6d6b5f27fb1d2daf80a2dcfe1a not found: ID does not exist"
May 20 16:36:42.895783 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.895762    2556 scope.go:117] "RemoveContainer" containerID="6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb"
May 20 16:36:42.895946 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:36:42.895924    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb\": container with ID starting with 6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb not found: ID does not exist" containerID="6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb"
May 20 16:36:42.895988 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.895954    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb"} err="failed to get container status \"6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb\": rpc error: code = NotFound desc = could not find container \"6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb\": container with ID starting with 6904cfea3427532b6aabfc6189906389767a77a3baba9d379a8e5c81ef33e7bb not found: ID does not exist"
May 20 16:36:42.895988 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.895969    2556 scope.go:117] "RemoveContainer" containerID="0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34"
May 20 16:36:42.896154 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:36:42.896139    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34\": container with ID starting with 0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34 not found: ID does not exist" containerID="0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34"
May 20 16:36:42.896202 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.896157    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34"} err="failed to get container status \"0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34\": rpc error: code = NotFound desc = could not find container \"0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34\": container with ID starting with 0e644cf2b42e1e72e8f86585d502818d89f6d69c78dd9442fd23ab79e3122a34 not found: ID does not exist"
May 20 16:36:42.899404 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899387    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b353fb0-a562-4d67-8377-b682f28a1e81-kserve-provision-location\") pod \"4b353fb0-a562-4d67-8377-b682f28a1e81\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") "
May 20 16:36:42.899486 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899425    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"4b353fb0-a562-4d67-8377-b682f28a1e81\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") "
May 20 16:36:42.899486 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899478    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b353fb0-a562-4d67-8377-b682f28a1e81-proxy-tls\") pod \"4b353fb0-a562-4d67-8377-b682f28a1e81\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") "
May 20 16:36:42.899605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899533    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js46b\" (UniqueName: \"kubernetes.io/projected/4b353fb0-a562-4d67-8377-b682f28a1e81-kube-api-access-js46b\") pod \"4b353fb0-a562-4d67-8377-b682f28a1e81\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") "
May 20 16:36:42.899605 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899588    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-cabundle-cert\") pod \"4b353fb0-a562-4d67-8377-b682f28a1e81\" (UID: \"4b353fb0-a562-4d67-8377-b682f28a1e81\") "
May 20 16:36:42.899710 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899692    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b353fb0-a562-4d67-8377-b682f28a1e81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b353fb0-a562-4d67-8377-b682f28a1e81" (UID: "4b353fb0-a562-4d67-8377-b682f28a1e81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:36:42.899843 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899816    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "4b353fb0-a562-4d67-8377-b682f28a1e81" (UID: "4b353fb0-a562-4d67-8377-b682f28a1e81"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:36:42.899843 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899838    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b353fb0-a562-4d67-8377-b682f28a1e81-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:42.899943 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.899906    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b353fb0-a562-4d67-8377-b682f28a1e81" (UID: "4b353fb0-a562-4d67-8377-b682f28a1e81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:36:42.901297 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.901278    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b353fb0-a562-4d67-8377-b682f28a1e81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b353fb0-a562-4d67-8377-b682f28a1e81" (UID: "4b353fb0-a562-4d67-8377-b682f28a1e81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:36:42.901562 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:42.901539    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b353fb0-a562-4d67-8377-b682f28a1e81-kube-api-access-js46b" (OuterVolumeSpecName: "kube-api-access-js46b") pod "4b353fb0-a562-4d67-8377-b682f28a1e81" (UID: "4b353fb0-a562-4d67-8377-b682f28a1e81"). InnerVolumeSpecName "kube-api-access-js46b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:36:43.000263 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:43.000240    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js46b\" (UniqueName: \"kubernetes.io/projected/4b353fb0-a562-4d67-8377-b682f28a1e81-kube-api-access-js46b\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:43.000263 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:43.000264    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-cabundle-cert\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:43.000387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:43.000274    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b353fb0-a562-4d67-8377-b682f28a1e81-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:43.000387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:43.000284    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b353fb0-a562-4d67-8377-b682f28a1e81-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:43.197300 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:43.197273    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"]
May 20 16:36:43.200267 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:43.200240    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-784cfcb679-nv6zs"]
May 20 16:36:43.891056 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:43.891018    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" path="/var/lib/kubelet/pods/4b353fb0-a562-4d67-8377-b682f28a1e81/volumes"
May 20 16:36:44.885401 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:44.885327    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_a67aed04-469c-4eb5-855b-68b9196d6210/storage-initializer/0.log"
May 20 16:36:44.885401 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:44.885369    2556 generic.go:358] "Generic (PLEG): container finished" podID="a67aed04-469c-4eb5-855b-68b9196d6210" containerID="7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef" exitCode=1
May 20 16:36:44.885586 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:44.885466    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn" event={"ID":"a67aed04-469c-4eb5-855b-68b9196d6210","Type":"ContainerDied","Data":"7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef"}
May 20 16:36:45.890310 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:45.890282    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_a67aed04-469c-4eb5-855b-68b9196d6210/storage-initializer/0.log"
May 20 16:36:45.890785 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:45.890405    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn" event={"ID":"a67aed04-469c-4eb5-855b-68b9196d6210","Type":"ContainerStarted","Data":"57f3191ed79f84ff76a380daa579640ec3c9bf70f4f7fbb7075f3f83c9532e89"}
May 20 16:36:48.900420 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:48.900389    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_a67aed04-469c-4eb5-855b-68b9196d6210/storage-initializer/1.log"
May 20 16:36:48.900936 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:48.900917    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_a67aed04-469c-4eb5-855b-68b9196d6210/storage-initializer/0.log"
May 20 16:36:48.900991 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:48.900957    2556 generic.go:358] "Generic (PLEG): container finished" podID="a67aed04-469c-4eb5-855b-68b9196d6210" containerID="57f3191ed79f84ff76a380daa579640ec3c9bf70f4f7fbb7075f3f83c9532e89" exitCode=1
May 20 16:36:48.901029 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:48.901005    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn" event={"ID":"a67aed04-469c-4eb5-855b-68b9196d6210","Type":"ContainerDied","Data":"57f3191ed79f84ff76a380daa579640ec3c9bf70f4f7fbb7075f3f83c9532e89"}
May 20 16:36:48.901067 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:48.901041    2556 scope.go:117] "RemoveContainer" containerID="7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef"
May 20 16:36:48.901522 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:48.901493    2556 scope.go:117] "RemoveContainer" containerID="7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef"
May 20 16:36:48.912286 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:36:48.912242    2556 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_kserve-ci-e2e-test_a67aed04-469c-4eb5-855b-68b9196d6210_0 in pod sandbox 521dd0af690cb02efca43e532f34c480513f998ac8b00565be577b8624733e99 from index: no such id: '7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef'" containerID="7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef"
May 20 16:36:48.912428 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:48.912292    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_kserve-ci-e2e-test_a67aed04-469c-4eb5-855b-68b9196d6210_0 in pod sandbox 521dd0af690cb02efca43e532f34c480513f998ac8b00565be577b8624733e99 from index: no such id: '7d11073de18f3ecee8a26a60276185e24316a6f8e9875f7a12b152474d0aa9ef'"
May 20 16:36:48.912485 ip-10-0-134-233 kubenswrapper[2556]: E0520 16:36:48.912427    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_kserve-ci-e2e-test(a67aed04-469c-4eb5-855b-68b9196d6210)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn" podUID="a67aed04-469c-4eb5-855b-68b9196d6210"
May 20 16:36:49.904926 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:49.904899    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_a67aed04-469c-4eb5-855b-68b9196d6210/storage-initializer/1.log"
May 20 16:36:50.210144 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.210048    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"]
May 20 16:36:50.328399 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.328380    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_a67aed04-469c-4eb5-855b-68b9196d6210/storage-initializer/1.log"
May 20 16:36:50.328501 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.328440    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:50.452274 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.452248    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a67aed04-469c-4eb5-855b-68b9196d6210-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"a67aed04-469c-4eb5-855b-68b9196d6210\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") "
May 20 16:36:50.452387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.452285    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a67aed04-469c-4eb5-855b-68b9196d6210-kserve-provision-location\") pod \"a67aed04-469c-4eb5-855b-68b9196d6210\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") "
May 20 16:36:50.452387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.452312    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a67aed04-469c-4eb5-855b-68b9196d6210-proxy-tls\") pod \"a67aed04-469c-4eb5-855b-68b9196d6210\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") "
May 20 16:36:50.452387 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.452374    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/a67aed04-469c-4eb5-855b-68b9196d6210-kube-api-access-2kmhw\") pod \"a67aed04-469c-4eb5-855b-68b9196d6210\" (UID: \"a67aed04-469c-4eb5-855b-68b9196d6210\") "
May 20 16:36:50.452572 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.452552    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67aed04-469c-4eb5-855b-68b9196d6210-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a67aed04-469c-4eb5-855b-68b9196d6210" (UID: "a67aed04-469c-4eb5-855b-68b9196d6210"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:36:50.452656 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.452637    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67aed04-469c-4eb5-855b-68b9196d6210-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "a67aed04-469c-4eb5-855b-68b9196d6210" (UID: "a67aed04-469c-4eb5-855b-68b9196d6210"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 16:36:50.454385 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.454358    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67aed04-469c-4eb5-855b-68b9196d6210-kube-api-access-2kmhw" (OuterVolumeSpecName: "kube-api-access-2kmhw") pod "a67aed04-469c-4eb5-855b-68b9196d6210" (UID: "a67aed04-469c-4eb5-855b-68b9196d6210"). InnerVolumeSpecName "kube-api-access-2kmhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:36:50.454489 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.454420    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67aed04-469c-4eb5-855b-68b9196d6210-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a67aed04-469c-4eb5-855b-68b9196d6210" (UID: "a67aed04-469c-4eb5-855b-68b9196d6210"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:36:50.552815 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.552779    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a67aed04-469c-4eb5-855b-68b9196d6210-kserve-provision-location\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:50.552815 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.552816    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a67aed04-469c-4eb5-855b-68b9196d6210-proxy-tls\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:50.552963 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.552828    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/a67aed04-469c-4eb5-855b-68b9196d6210-kube-api-access-2kmhw\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:50.552963 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.552838    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a67aed04-469c-4eb5-855b-68b9196d6210-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-233.ec2.internal\" DevicePath \"\""
May 20 16:36:50.908954 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.908892    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn_a67aed04-469c-4eb5-855b-68b9196d6210/storage-initializer/1.log"
May 20 16:36:50.909339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.908965    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn" event={"ID":"a67aed04-469c-4eb5-855b-68b9196d6210","Type":"ContainerDied","Data":"521dd0af690cb02efca43e532f34c480513f998ac8b00565be577b8624733e99"}
May 20 16:36:50.909339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.909002    2556 scope.go:117] "RemoveContainer" containerID="57f3191ed79f84ff76a380daa579640ec3c9bf70f4f7fbb7075f3f83c9532e89"
May 20 16:36:50.909339 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.909084    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"
May 20 16:36:50.947132 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.947084    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"]
May 20 16:36:50.950785 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:50.950764    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d9b88c469-75sbn"]
May 20 16:36:51.891054 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:36:51.891008    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a67aed04-469c-4eb5-855b-68b9196d6210" path="/var/lib/kubelet/pods/a67aed04-469c-4eb5-855b-68b9196d6210/volumes"
May 20 16:37:23.674925 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.674842    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mwsjl/must-gather-wzf8h"]
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675167    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675182    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675200    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67aed04-469c-4eb5-855b-68b9196d6210" containerName="storage-initializer"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675208    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67aed04-469c-4eb5-855b-68b9196d6210" containerName="storage-initializer"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675217    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kube-rbac-proxy"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675223    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kube-rbac-proxy"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675232    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="storage-initializer"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675237    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="storage-initializer"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675293    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="a67aed04-469c-4eb5-855b-68b9196d6210" containerName="storage-initializer"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675304    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kube-rbac-proxy"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675312    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b353fb0-a562-4d67-8377-b682f28a1e81" containerName="kserve-container"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675346    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67aed04-469c-4eb5-855b-68b9196d6210" containerName="storage-initializer"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675352    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67aed04-469c-4eb5-855b-68b9196d6210" containerName="storage-initializer"
May 20 16:37:23.675438 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.675400    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="a67aed04-469c-4eb5-855b-68b9196d6210" containerName="storage-initializer"
May 20 16:37:23.678158 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.678141    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwsjl/must-gather-wzf8h"
May 20 16:37:23.681051 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.681026    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mwsjl\"/\"openshift-service-ca.crt\""
May 20 16:37:23.681237 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.681067    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mwsjl\"/\"kube-root-ca.crt\""
May 20 16:37:23.682120 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.682075    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mwsjl\"/\"default-dockercfg-b4sdk\""
May 20 16:37:23.684914 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.684890    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwsjl/must-gather-wzf8h"]
May 20 16:37:23.781404 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.781380    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57443ef1-15fc-4ddc-aec7-49229bb84aa5-must-gather-output\") pod \"must-gather-wzf8h\" (UID: \"57443ef1-15fc-4ddc-aec7-49229bb84aa5\") " pod="openshift-must-gather-mwsjl/must-gather-wzf8h"
May 20 16:37:23.781525 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.781433    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvlz\" (UniqueName: \"kubernetes.io/projected/57443ef1-15fc-4ddc-aec7-49229bb84aa5-kube-api-access-vtvlz\") pod \"must-gather-wzf8h\" (UID: \"57443ef1-15fc-4ddc-aec7-49229bb84aa5\") " pod="openshift-must-gather-mwsjl/must-gather-wzf8h"
May 20 16:37:23.882158 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.882133    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvlz\" (UniqueName: \"kubernetes.io/projected/57443ef1-15fc-4ddc-aec7-49229bb84aa5-kube-api-access-vtvlz\") pod \"must-gather-wzf8h\" (UID: \"57443ef1-15fc-4ddc-aec7-49229bb84aa5\") " pod="openshift-must-gather-mwsjl/must-gather-wzf8h"
May 20 16:37:23.882266 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.882174    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57443ef1-15fc-4ddc-aec7-49229bb84aa5-must-gather-output\") pod \"must-gather-wzf8h\" (UID: \"57443ef1-15fc-4ddc-aec7-49229bb84aa5\") " pod="openshift-must-gather-mwsjl/must-gather-wzf8h"
May 20 16:37:23.882428 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.882412    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57443ef1-15fc-4ddc-aec7-49229bb84aa5-must-gather-output\") pod \"must-gather-wzf8h\" (UID: \"57443ef1-15fc-4ddc-aec7-49229bb84aa5\") " pod="openshift-must-gather-mwsjl/must-gather-wzf8h"
May 20 16:37:23.890992 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.890972    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvlz\" (UniqueName: \"kubernetes.io/projected/57443ef1-15fc-4ddc-aec7-49229bb84aa5-kube-api-access-vtvlz\") pod \"must-gather-wzf8h\" (UID: \"57443ef1-15fc-4ddc-aec7-49229bb84aa5\") " pod="openshift-must-gather-mwsjl/must-gather-wzf8h"
May 20 16:37:23.988088 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:23.988068    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwsjl/must-gather-wzf8h"
May 20 16:37:24.114220 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:24.114190    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwsjl/must-gather-wzf8h"]
May 20 16:37:24.117790 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:37:24.117764    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57443ef1_15fc_4ddc_aec7_49229bb84aa5.slice/crio-c26f9c77327c83ff9b31ed82fb350f9c4d4d213fc4a6d1d10d75b0d1ec3c4362 WatchSource:0}: Error finding container c26f9c77327c83ff9b31ed82fb350f9c4d4d213fc4a6d1d10d75b0d1ec3c4362: Status 404 returned error can't find the container with id c26f9c77327c83ff9b31ed82fb350f9c4d4d213fc4a6d1d10d75b0d1ec3c4362
May 20 16:37:25.006983 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:25.006947    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwsjl/must-gather-wzf8h" event={"ID":"57443ef1-15fc-4ddc-aec7-49229bb84aa5","Type":"ContainerStarted","Data":"c26f9c77327c83ff9b31ed82fb350f9c4d4d213fc4a6d1d10d75b0d1ec3c4362"}
May 20 16:37:26.011733 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:26.011699    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwsjl/must-gather-wzf8h" event={"ID":"57443ef1-15fc-4ddc-aec7-49229bb84aa5","Type":"ContainerStarted","Data":"5a9b5d281591a341764dda5f732b70e1c92c92697a8bba01f923641f3ab0c6ff"}
May 20 16:37:26.012139 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:26.011741    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwsjl/must-gather-wzf8h" event={"ID":"57443ef1-15fc-4ddc-aec7-49229bb84aa5","Type":"ContainerStarted","Data":"3e4ba1a8e646b8a231778014ad86d498685273ca56cf94a12def49c19b28cc50"}
May 20 16:37:26.027649 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:26.027594    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-mwsjl/must-gather-wzf8h" podStartSLOduration=2.041049347 podStartE2EDuration="3.027579381s" podCreationTimestamp="2026-05-20 16:37:23 +0000 UTC" firstStartedPulling="2026-05-20 16:37:24.119982958 +0000 UTC m=+3886.811238865" lastFinishedPulling="2026-05-20 16:37:25.106512992 +0000 UTC m=+3887.797768899" observedRunningTime="2026-05-20 16:37:26.026883839 +0000 UTC m=+3888.718139766" watchObservedRunningTime="2026-05-20 16:37:26.027579381 +0000 UTC m=+3888.718835329"
May 20 16:37:26.606553 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:26.606514    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9lt8j_45de9667-3479-4085-8ff9-bbc7b04205c2/global-pull-secret-syncer/0.log"
May 20 16:37:26.763636 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:26.763602    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8qs26_96a608c3-26ba-4e9a-935b-253eecb12586/konnectivity-agent/0.log"
May 20 16:37:26.909770 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:26.909698    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-233.ec2.internal_1258e990010ed0305514767d059e0701/haproxy/0.log"
May 20 16:37:30.383110 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:30.383068    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c85b47f9b-g4bkd_6dc91e19-7fb8-4872-b89d-f5dc48522d58/metrics-server/0.log"
May 20 16:37:30.528551 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:30.528522    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfsrc_31a3434f-9205-4ec2-bb67-e0259aeb1a85/node-exporter/0.log"
May 20 16:37:30.551977 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:30.551946    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfsrc_31a3434f-9205-4ec2-bb67-e0259aeb1a85/kube-rbac-proxy/0.log"
May 20 16:37:30.577892 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:30.577843    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfsrc_31a3434f-9205-4ec2-bb67-e0259aeb1a85/init-textfile/0.log"
May 20 16:37:33.142475 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.142449    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bdc794bc6-gdfr9_90f39d67-2aee-42c2-b935-2bfd1eae06bc/console/0.log"
May 20 16:37:33.708830 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.708802    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"]
May 20 16:37:33.712682 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.712664    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.729182 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.729156    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"]
May 20 16:37:33.766796 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.766769    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-sys\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.766941 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.766816    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-podres\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.766941 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.766845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-lib-modules\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.766941 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.766878    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwxh\" (UniqueName: \"kubernetes.io/projected/2a2d990c-8077-4681-9cf4-601d6b81d982-kube-api-access-zjwxh\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.767069 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.766959    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-proc\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.867890 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.867854    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwxh\" (UniqueName: \"kubernetes.io/projected/2a2d990c-8077-4681-9cf4-601d6b81d982-kube-api-access-zjwxh\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.868054 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.867936    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-proc\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.868054 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.867967    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-sys\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.868054 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.867998    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-podres\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.868054 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.868025    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-lib-modules\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.868292 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.868068    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-sys\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.868292 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.868066    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-proc\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.868292 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.868200    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-podres\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.868292 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.868221    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a2d990c-8077-4681-9cf4-601d6b81d982-lib-modules\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:33.876933 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:33.876904    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwxh\" (UniqueName: \"kubernetes.io/projected/2a2d990c-8077-4681-9cf4-601d6b81d982-kube-api-access-zjwxh\") pod \"perf-node-gather-daemonset-r88v2\" (UID: \"2a2d990c-8077-4681-9cf4-601d6b81d982\") " pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:34.021774 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:34.021729    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:34.151682 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:34.151653    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"]
May 20 16:37:34.157944 ip-10-0-134-233 kubenswrapper[2556]: W0520 16:37:34.157906    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2a2d990c_8077_4681_9cf4_601d6b81d982.slice/crio-4af951c7c6ee5c15384cd4c483af5e4d5650c116ff9191bae6be5b3e101d3a0c WatchSource:0}: Error finding container 4af951c7c6ee5c15384cd4c483af5e4d5650c116ff9191bae6be5b3e101d3a0c: Status 404 returned error can't find the container with id 4af951c7c6ee5c15384cd4c483af5e4d5650c116ff9191bae6be5b3e101d3a0c
May 20 16:37:34.293975 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:34.293908    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4z29q_b9bbd465-a5f9-43a2-b969-0f6be7230e22/dns/0.log"
May 20 16:37:34.317198 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:34.317178    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4z29q_b9bbd465-a5f9-43a2-b969-0f6be7230e22/kube-rbac-proxy/0.log"
May 20 16:37:34.497832 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:34.497804    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rs4nk_f95984e7-eb1b-431d-9b72-e34522b8e80f/dns-node-resolver/0.log"
May 20 16:37:34.935921 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:34.935891    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67948b89b7-mm82h_4a35d922-49ce-401d-85e3-cafb1b11f072/registry/0.log"
May 20 16:37:34.962847 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:34.962821    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dw68c_ea0f63ed-dc2b-42b1-8ad6-b83b95c52940/node-ca/0.log"
May 20 16:37:35.043773 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:35.043738    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2" event={"ID":"2a2d990c-8077-4681-9cf4-601d6b81d982","Type":"ContainerStarted","Data":"1aa7d1cbc3adae118716d751dd8a2da836307fb7d0ca68492aa0542703bea39d"}
May 20 16:37:35.043938 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:35.043780    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2" event={"ID":"2a2d990c-8077-4681-9cf4-601d6b81d982","Type":"ContainerStarted","Data":"4af951c7c6ee5c15384cd4c483af5e4d5650c116ff9191bae6be5b3e101d3a0c"}
May 20 16:37:35.044418 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:35.044397    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:35.062072 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:35.062010    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2" podStartSLOduration=2.061994004 podStartE2EDuration="2.061994004s" podCreationTimestamp="2026-05-20 16:37:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:37:35.06146223 +0000 UTC m=+3897.752718148" watchObservedRunningTime="2026-05-20 16:37:35.061994004 +0000 UTC m=+3897.753249934"
May 20 16:37:36.065618 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:36.065586    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9xjr_a46d22fb-89c2-4738-bdce-892fb1a3fd53/serve-healthcheck-canary/0.log"
May 20 16:37:36.612846 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:36.612801    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbm2j_df9c9219-d33c-4426-8aa2-d340b4cc6559/kube-rbac-proxy/0.log"
May 20 16:37:36.633485 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:36.633454    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbm2j_df9c9219-d33c-4426-8aa2-d340b4cc6559/exporter/0.log"
May 20 16:37:36.654900 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:36.654876    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbm2j_df9c9219-d33c-4426-8aa2-d340b4cc6559/extractor/0.log"
May 20 16:37:38.677974 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:38.677946    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-68cc5db7c4-xwx29_01a55bae-a072-4d7d-ada8-9f1670136ef7/manager/0.log"
May 20 16:37:42.060585 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:42.060555    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mwsjl/perf-node-gather-daemonset-r88v2"
May 20 16:37:44.270285 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.270259    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cn9s_72179c2e-fdb0-4e86-86ec-7be1ff47ea67/kube-multus-additional-cni-plugins/0.log"
May 20 16:37:44.294048 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.293977    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cn9s_72179c2e-fdb0-4e86-86ec-7be1ff47ea67/egress-router-binary-copy/0.log"
May 20 16:37:44.317407 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.317375    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cn9s_72179c2e-fdb0-4e86-86ec-7be1ff47ea67/cni-plugins/0.log"
May 20 16:37:44.343566 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.343526    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cn9s_72179c2e-fdb0-4e86-86ec-7be1ff47ea67/bond-cni-plugin/0.log"
May 20 16:37:44.369421 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.369388    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cn9s_72179c2e-fdb0-4e86-86ec-7be1ff47ea67/routeoverride-cni/0.log"
May 20 16:37:44.396340 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.396306    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cn9s_72179c2e-fdb0-4e86-86ec-7be1ff47ea67/whereabouts-cni-bincopy/0.log"
May 20 16:37:44.420238 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.420212    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cn9s_72179c2e-fdb0-4e86-86ec-7be1ff47ea67/whereabouts-cni/0.log"
May 20 16:37:44.833637 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.833607    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z98x6_f82c6669-7244-4512-aaa5-344d379dd5c5/kube-multus/0.log"
May 20 16:37:44.856269 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.856248    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-44pnc_e67e163d-ec7d-4d41-b2b6-27e8bbab3825/network-metrics-daemon/0.log"
May 20 16:37:44.873567 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:44.873549    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-44pnc_e67e163d-ec7d-4d41-b2b6-27e8bbab3825/kube-rbac-proxy/0.log"
May 20 16:37:46.117506 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:46.117476    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdrhl_4e273678-1148-4636-a24d-895144217e21/ovn-controller/0.log"
May 20 16:37:46.170388 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:46.170362    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdrhl_4e273678-1148-4636-a24d-895144217e21/ovn-acl-logging/0.log"
May 20 16:37:46.191982 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:46.191951    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdrhl_4e273678-1148-4636-a24d-895144217e21/kube-rbac-proxy-node/0.log"
May 20 16:37:46.215599 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:46.215579    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdrhl_4e273678-1148-4636-a24d-895144217e21/kube-rbac-proxy-ovn-metrics/0.log"
May 20 16:37:46.231529 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:46.231500    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdrhl_4e273678-1148-4636-a24d-895144217e21/northd/0.log"
May 20 16:37:46.251495 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:46.251475    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdrhl_4e273678-1148-4636-a24d-895144217e21/nbdb/0.log"
May 20 16:37:46.271898 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:46.271873    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdrhl_4e273678-1148-4636-a24d-895144217e21/sbdb/0.log"
May 20 16:37:46.458608 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:46.458543    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hdrhl_4e273678-1148-4636-a24d-895144217e21/ovnkube-controller/0.log"
May 20 16:37:47.774317 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:47.774295    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8c6s4_b909f61b-515c-409b-9fa7-8ae2dea1e342/network-check-target-container/0.log"
May 20 16:37:48.748892 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:48.748867    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-tqf9l_c81f7e0c-18a6-4c32-931e-852550317209/iptables-alerter/0.log"
May 20 16:37:49.389590 ip-10-0-134-233 kubenswrapper[2556]: I0520 16:37:49.389511    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mpbk4_eca1dc24-4845-4c65-a1ea-765efbdec276/tuned/0.log"