May 26 11:27:17.515139 ip-10-0-138-207 systemd[1]: Starting Kubernetes Kubelet...
May 26 11:27:17.983651 ip-10-0-138-207 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:27:17.983651 ip-10-0-138-207 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 11:27:17.983651 ip-10-0-138-207 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:27:17.983651 ip-10-0-138-207 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 11:27:17.983651 ip-10-0-138-207 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:27:17.985513 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.985430    2563 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 11:27:17.990141 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990128    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:27:17.990141 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990142    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990145    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990148    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990151    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990155    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990158    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990161    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990164    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990166    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990169    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990172    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990179    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990182    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990184    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990187    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990189    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990192    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990194    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990197    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990199    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:27:17.990200 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990202    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990205    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990207    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990211    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990215    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990218    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990221    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990223    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990226    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990229    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990232    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990234    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990236    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990239    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990241    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990243    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990246    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990248    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990251    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990253    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:27:17.990649 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990255    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990258    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990268    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990271    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990273    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990275    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990278    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990280    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990282    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990285    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990287    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990290    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990293    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990296    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990299    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990302    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990305    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990307    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990310    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:27:17.991125 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990312    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990315    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990318    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990320    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990322    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990325    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990327    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990330    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990332    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990336    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990339    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990342    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990345    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990347    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990349    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990352    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990359    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990362    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990365    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990368    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:27:17.991578 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990370    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990372    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990375    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990377    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990380    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990382    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990819    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990826    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990829    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990832    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990834    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990837    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990840    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990842    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990845    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990847    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990850    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990853    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990856    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990858    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:27:17.992078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990860    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990863    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990866    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990868    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990871    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990873    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990875    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990878    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990887    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990890    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990892    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990894    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990896    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990899    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990902    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990904    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990907    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990909    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990912    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990915    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:27:17.992969 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990918    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990920    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990922    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990925    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990928    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990930    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990932    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990934    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990937    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990939    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990942    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990944    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990946    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990949    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990951    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990954    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990956    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990958    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990960    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:27:17.993471 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990963    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990967    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990976    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990980    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990983    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990986    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990990    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990993    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990996    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.990999    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991002    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991004    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991008    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991010    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991013    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991016    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991019    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991021    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991024    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:27:17.993935 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991027    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991030    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991033    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991035    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991038    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991041    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991044    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991047    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991049    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991052    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991054    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991056    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991059    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.991061    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.992018    2563 flags.go:64] FLAG: --address="0.0.0.0"
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.992028    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.992073    2563 flags.go:64] FLAG: --anonymous-auth="true"
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993170    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993354    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993361    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993374    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 11:27:17.994395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993379    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993382    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993385    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993389    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993392    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993395    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993398    2563 flags.go:64] FLAG: --cgroup-root=""
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993401    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993404    2563 flags.go:64] FLAG: --client-ca-file=""
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993407    2563 flags.go:64] FLAG: --cloud-config=""
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993409    2563 flags.go:64] FLAG: --cloud-provider="external"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993412    2563 flags.go:64] FLAG: --cluster-dns="[]"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993419    2563 flags.go:64] FLAG: --cluster-domain=""
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993422    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993425    2563 flags.go:64] FLAG: --config-dir=""
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993428    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993431    2563 flags.go:64] FLAG: --container-log-max-files="5"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993435    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993438    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993441    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993444    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993447    2563 flags.go:64] FLAG: --contention-profiling="false"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993450    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993454    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993457    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 11:27:17.994911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993460    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993464    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993467    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993470    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993474    2563 flags.go:64] FLAG: --enable-load-reader="false"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993477    2563 flags.go:64] FLAG: --enable-server="true"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993480    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993485    2563 flags.go:64] FLAG: --event-burst="100"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993488    2563 flags.go:64] FLAG: --event-qps="50"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993491    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993494    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993496    2563 flags.go:64] FLAG: --eviction-hard=""
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993500    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993503    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993506    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993509    2563 flags.go:64] FLAG: --eviction-soft=""
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993512    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993514    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993517    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993520    2563 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993523    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993526    2563 flags.go:64] FLAG: --fail-swap-on="true"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993529    2563 flags.go:64] FLAG: --feature-gates=""
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993533    2563 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993536    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 11:27:17.995490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993539    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993542    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993545    2563 flags.go:64] FLAG: --healthz-port="10248"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993548    2563 flags.go:64] FLAG: --help="false"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993551    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-138-207.ec2.internal"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993554    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993557    2563 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993560    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993563    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993568    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993571    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993575    2563 flags.go:64] FLAG: --image-service-endpoint=""
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993577    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993580    2563 flags.go:64] FLAG: --kube-api-burst="100"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993583    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993586    2563 flags.go:64] FLAG: --kube-api-qps="50"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993589    2563 flags.go:64] FLAG: --kube-reserved=""
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993591    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993594    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993597    2563 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993599    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993602    2563 flags.go:64] FLAG: --lock-file=""
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993605    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993607    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 11:27:17.996110 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993610    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993615    2563 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993618    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993621    2563 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993623    2563 flags.go:64] FLAG: --logging-format="text"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993626    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993630    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993632    2563 flags.go:64] FLAG: --manifest-url=""
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993635    2563 flags.go:64] FLAG: --manifest-url-header=""
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993640    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993643    2563 flags.go:64] FLAG: --max-open-files="1000000"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993647    2563 flags.go:64] FLAG: --max-pods="110"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993650    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993653    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993655    2563 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993658    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993661    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993664    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993667    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993675    2563 flags.go:64] FLAG: --node-status-max-images="50"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993679    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993682    2563 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993684    2563 flags.go:64] FLAG: --pod-cidr=""
May 26 11:27:17.996705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993687    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993692    2563 flags.go:64] FLAG: --pod-manifest-path=""
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993695    2563 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993698    2563 flags.go:64] FLAG: --pods-per-core="0"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993701    2563 flags.go:64] FLAG: --port="10250"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993704    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993706    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0abb23f0537057008"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993709    2563 flags.go:64] FLAG: --qos-reserved=""
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993713    2563 flags.go:64] FLAG: --read-only-port="10255"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993717    2563 flags.go:64] FLAG: --register-node="true"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993720    2563 flags.go:64] FLAG: --register-schedulable="true"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993723    2563 flags.go:64] FLAG: --register-with-taints=""
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993726    2563 flags.go:64] FLAG: --registry-burst="10"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993730    2563 flags.go:64] FLAG: --registry-qps="5"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993732    2563 flags.go:64] FLAG: --reserved-cpus=""
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993735    2563 flags.go:64] FLAG: --reserved-memory=""
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993739    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993742    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993745    2563 flags.go:64] FLAG: --rotate-certificates="false"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993748    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993750    2563 flags.go:64] FLAG: --runonce="false"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993753    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993756    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993760    2563 flags.go:64] FLAG: --seccomp-default="false"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993763    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993766    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 11:27:17.997296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993769    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993772    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993775    2563 flags.go:64] FLAG: --storage-driver-password="root"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993778    2563 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993781    2563 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993784    2563 flags.go:64] FLAG: --storage-driver-user="root"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993786    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993803    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993806    2563 flags.go:64] FLAG: --system-cgroups=""
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993809    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993815    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993818    2563 flags.go:64] FLAG: --tls-cert-file=""
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993820    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993825    2563 flags.go:64] FLAG: --tls-min-version=""
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993828    2563 flags.go:64] FLAG: --tls-private-key-file=""
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993830    2563 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993834    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993837    2563 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993839    2563 flags.go:64] FLAG: --v="2"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993843    2563 flags.go:64] FLAG: --version="false"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993847    2563 flags.go:64] FLAG: --vmodule=""
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993852    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.993855    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993950    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:27:17.997932 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993954    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993957    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993959    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993962    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993965    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993968    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993971    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993974    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993978    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993980    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993983    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993986    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993989    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993991    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993994    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993996    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.993999    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994001    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994003    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994006    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:27:17.998481 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994008    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994011    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994013    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994016    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994018    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994021    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994023    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994026    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994028    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994030    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994033    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994035    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994037    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994040    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994044    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994047    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994055    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994058    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994060    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:27:17.998971 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994062    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994065    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994068    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994072    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994074    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994077    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994079    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994082    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994084    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994087    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994089    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994092    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994094    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994097    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994099    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994102    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994105    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994109    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994112    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994116    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:27:17.999408 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994118    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994121    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994124    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994126    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994129    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994131    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994133    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994136    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994139    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994143    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994146    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994149    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994151    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994154    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994157    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994160    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994163    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994166    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994169    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994171    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:27:17.999885 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994174    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:27:18.000406 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994176    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:27:18.000406 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994179    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:27:18.000406 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994181    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:27:18.000406 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994184    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:27:18.000406 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:17.994187    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:27:18.000406 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:17.994192    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:27:18.001547 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.001530    2563 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 11:27:18.001579 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.001548    2563 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 11:27:18.001605 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001594    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:27:18.001605 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001600    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:27:18.001605 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001606    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001610    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001613    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001616    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001619    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001622    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001625    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001627    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001630    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001632    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001635    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001637    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001639    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001642    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001645    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001647    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001650    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001652    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001655    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001657    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:27:18.001690 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001660    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001663    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001665    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001667    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001670    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001672    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001675    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001677    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001680    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001682    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001685    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001687    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001690    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001694    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001697    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001699    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001702    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001705    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001707    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001710    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:27:18.002211 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001712    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001715    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001717    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001720    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001722    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001724    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001727    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001729    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001731    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001734    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001736    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001739    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001741    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001744    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001748    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001750    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001753    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001756    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001758    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001760    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:27:18.002676 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001763    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001766    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001768    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001771    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001773    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001775    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001778    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001781    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001783    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001786    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001804    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001808    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001811    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001814    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001816    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001819    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001821    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001824    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001826    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:27:18.003173 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001828    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001832    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001836    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001839    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001841    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.001847    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001959    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001964    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001967    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001970    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001973    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001976    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001979    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001981    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001985    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:27:18.003604 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001987    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001990    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001992    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001995    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.001997    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002000    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002003    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002005    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002008    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002010    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002013    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002016    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002018    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002021    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002023    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002026    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002028    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002030    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002032    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002035    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:27:18.003996 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002037    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002040    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002042    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002057    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002060    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002063    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002065    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002067    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002070    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002072    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002075    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002077    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002079    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002082    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002084    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002086    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002089    2563 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002091    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002094    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:27:18.004477 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002097    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002099    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002102    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002104    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002106    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002108    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002111    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002113    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002115    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002118    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002120    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002123    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002126    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002128    2563 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002130    2563 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002133    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002135    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002138    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002140    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002142    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:27:18.004942 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002145    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002148    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002151    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002154    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002156    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002159    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002161    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002164    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002167    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002169    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002172    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002175    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002182    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002185    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002188    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002191    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002194    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:27:18.005417 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:18.002196    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:27:18.005841 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.002201    2563 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:27:18.005841 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.002988    2563 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 11:27:18.008076 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.008062    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 11:27:18.009173 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.009162    2563 server.go:1019] "Starting client certificate rotation"
May 26 11:27:18.009279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.009263    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:27:18.009307 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.009302    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:27:18.036103 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.036080    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:27:18.038005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.037988    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:27:18.053405 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.053387    2563 log.go:25] "Validated CRI v1 runtime API"
May 26 11:27:18.058348 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.058330    2563 log.go:25] "Validated CRI v1 image API"
May 26 11:27:18.059533 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.059519    2563 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 11:27:18.064421 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.064394    2563 fs.go:135] Filesystem UUIDs: map[70a0c537-4546-46bb-9401-c70423e414ea:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f5ee2d7b-cbae-4997-b08c-d20b50928a53:/dev/nvme0n1p4]
May 26 11:27:18.064504 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.064420    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 11:27:18.069202 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.069082    2563 manager.go:217] Machine: {Timestamp:2026-05-26 11:27:18.068542401 +0000 UTC m=+0.431492061 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3194795 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23e09011781feee5809d6367fb79ac SystemUUID:ec23e090-1178-1fee-e580-9d6367fb79ac BootID:1622597f-0400-480c-948c-ea15405f7a65 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:35:4e:e7:36:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:35:4e:e7:36:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:19:14:1f:ad:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 11:27:18.070847 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.070833    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 11:27:18.070928 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.070916    2563 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 11:27:18.072373 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.072336    2563 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 11:27:18.072506 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.072491    2563 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:27:18.072543 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.072368    2563 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-207.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 11:27:18.072543 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.072528    2563 topology_manager.go:138] "Creating topology manager with none policy"
May 26 11:27:18.072543 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.072538    2563 container_manager_linux.go:306] "Creating device plugin manager"
May 26 11:27:18.072649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.072548    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:27:18.074138 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.074127    2563 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:27:18.075820 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.075807    2563 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:27:18.075921 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.075912    2563 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 11:27:18.078532 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.078523    2563 kubelet.go:491] "Attempting to sync node with API server"
May 26 11:27:18.078563 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.078535    2563 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 11:27:18.078563 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.078547    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 11:27:18.078563 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.078555    2563 kubelet.go:397] "Adding apiserver pod source"
May 26 11:27:18.078563 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.078564    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 11:27:18.079807 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.079783    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:27:18.079864 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.079821    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:27:18.083021 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.083006    2563 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 11:27:18.085122 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.085109    2563 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 11:27:18.086081 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086068    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 11:27:18.086081 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086083    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086090    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086095    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086103    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086109    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086114    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086119    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086126    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086131    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 11:27:18.086209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086140    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 11:27:18.086458 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.086449    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 11:27:18.088711 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.088695    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 11:27:18.088758 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.088717    2563 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 11:27:18.090937 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.090914    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 11:27:18.090937 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.090927    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 11:27:18.091741 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.091729    2563 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 11:27:18.091805 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.091763    2563 server.go:1295] "Started kubelet"
May 26 11:27:18.091923 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.091838    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 11:27:18.092327 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.091923    2563 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 11:27:18.092389 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.092350    2563 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 11:27:18.092654 ip-10-0-138-207 systemd[1]: Started Kubernetes Kubelet.
May 26 11:27:18.093518 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.093477    2563 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 11:27:18.094074 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.094062    2563 server.go:317] "Adding debug handlers to kubelet server"
May 26 11:27:18.098967 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.098947    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 11:27:18.098967 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.098960    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 11:27:18.099100 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.099005    2563 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 11:27:18.099100 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.099016    2563 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 11:27:18.099196 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.099131    2563 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 11:27:18.099196 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.099168    2563 reconstruct.go:97] "Volume reconstruction finished"
May 26 11:27:18.099196 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.099176    2563 reconciler.go:26] "Reconciler: start to sync state"
May 26 11:27:18.099538 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.099522    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.102977 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.102956    2563 factory.go:55] Registering systemd factory
May 26 11:27:18.102977 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.102979    2563 factory.go:223] Registration of the systemd container factory successfully
May 26 11:27:18.105233 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.104935    2563 factory.go:153] Registering CRI-O factory
May 26 11:27:18.105233 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.104952    2563 factory.go:223] Registration of the crio container factory successfully
May 26 11:27:18.105233 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.105030    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 11:27:18.105233 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.105054    2563 factory.go:103] Registering Raw factory
May 26 11:27:18.105233 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.105071    2563 manager.go:1196] Started watching for new ooms in manager
May 26 11:27:18.105406 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.105398    2563 manager.go:319] Starting recovery of all containers
May 26 11:27:18.106370 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.106339    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 26 11:27:18.106447 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.106425    2563 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 26 11:27:18.106549 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.106535    2563 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-207.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 11:27:18.106682 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.106661    2563 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 11:27:18.107343 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.106316    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-207.ec2.internal.18b319f9b988383f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-207.ec2.internal,UID:ip-10-0-138-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-207.ec2.internal,},FirstTimestamp:2026-05-26 11:27:18.091741247 +0000 UTC m=+0.454690905,LastTimestamp:2026-05-26 11:27:18.091741247 +0000 UTC m=+0.454690905,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-207.ec2.internal,}"
May 26 11:27:18.109679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.109628    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nghb7"
May 26 11:27:18.114482 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.114466    2563 manager.go:324] Recovery completed
May 26 11:27:18.117541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.117527    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nghb7"
May 26 11:27:18.119210 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.119199    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:27:18.121337 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.121322    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:27:18.121420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.121350    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:27:18.121420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.121362    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientPID"
May 26 11:27:18.121891 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.121874    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 11:27:18.121891 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.121889    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 11:27:18.122007 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.121906    2563 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:27:18.123223 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.123159    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-207.ec2.internal.18b319f9bb4bd3ea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-207.ec2.internal,UID:ip-10-0-138-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-207.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-207.ec2.internal,},FirstTimestamp:2026-05-26 11:27:18.121337834 +0000 UTC m=+0.484287492,LastTimestamp:2026-05-26 11:27:18.121337834 +0000 UTC m=+0.484287492,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-207.ec2.internal,}"
May 26 11:27:18.124035 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.124021    2563 policy_none.go:49] "None policy: Start"
May 26 11:27:18.124096 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.124037    2563 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 11:27:18.124096 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.124048    2563 state_mem.go:35] "Initializing new in-memory state store"
May 26 11:27:18.183385 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.183369    2563 manager.go:341] "Starting Device Plugin manager"
May 26 11:27:18.183515 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.183411    2563 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 11:27:18.183515 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.183422    2563 server.go:85] "Starting device plugin registration server"
May 26 11:27:18.183677 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.183665    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 11:27:18.183726 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.183678    2563 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 11:27:18.183956 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.183778    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 11:27:18.184215 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.184202    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 11:27:18.184275 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.184218    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 11:27:18.185354 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.185335    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 11:27:18.185428 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.185390    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.236420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.236359    2563 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 11:27:18.237655 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.237643    2563 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 11:27:18.237703 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.237666    2563 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 11:27:18.237703 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.237683    2563 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 11:27:18.237703 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.237689    2563 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 11:27:18.237818 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.237753    2563 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 11:27:18.240326 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.240309    2563 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:27:18.284389 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.284364    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:27:18.288177 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.285160    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:27:18.288177 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.285189    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:27:18.288177 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.285199    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientPID"
May 26 11:27:18.288177 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.285220    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-207.ec2.internal"
May 26 11:27:18.296548 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.296528    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-207.ec2.internal"
May 26 11:27:18.296599 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.296553    2563 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-207.ec2.internal\": node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.316234 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.316213    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.338403 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.338382    2563 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal"]
May 26 11:27:18.338464 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.338451    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:27:18.339244 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.339226    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:27:18.339322 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.339254    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:27:18.339322 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.339265    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientPID"
May 26 11:27:18.340558 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.340547    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:27:18.340731 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.340718    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.340769 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.340745    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:27:18.341296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.341281    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:27:18.341348 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.341307    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:27:18.341348 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.341319    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientPID"
May 26 11:27:18.341408 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.341289    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:27:18.341408 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.341381    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:27:18.341408 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.341392    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientPID"
May 26 11:27:18.342479 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.342467    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.342521 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.342490    2563 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:27:18.343133 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.343115    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:27:18.343219 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.343142    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:27:18.343219 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.343152    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeHasSufficientPID"
May 26 11:27:18.367392 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.367371    2563 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-207.ec2.internal\" not found" node="ip-10-0-138-207.ec2.internal"
May 26 11:27:18.371492 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.371479    2563 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-207.ec2.internal\" not found" node="ip-10-0-138-207.ec2.internal"
May 26 11:27:18.401570 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.401549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e96ce5a1ede90039e45a1c264277675-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal\" (UID: \"3e96ce5a1ede90039e45a1c264277675\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.401642 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.401575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e96ce5a1ede90039e45a1c264277675-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal\" (UID: \"3e96ce5a1ede90039e45a1c264277675\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.401642 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.401600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6decc4be96339c77a5112ef19602d9b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-207.ec2.internal\" (UID: \"6decc4be96339c77a5112ef19602d9b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.416453 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.416435    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.502406 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.502336    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6decc4be96339c77a5112ef19602d9b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-207.ec2.internal\" (UID: \"6decc4be96339c77a5112ef19602d9b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.502406 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.502372    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e96ce5a1ede90039e45a1c264277675-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal\" (UID: \"3e96ce5a1ede90039e45a1c264277675\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.502406 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.502396    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e96ce5a1ede90039e45a1c264277675-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal\" (UID: \"3e96ce5a1ede90039e45a1c264277675\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.502556 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.502436    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6decc4be96339c77a5112ef19602d9b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-207.ec2.internal\" (UID: \"6decc4be96339c77a5112ef19602d9b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.502556 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.502437    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e96ce5a1ede90039e45a1c264277675-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal\" (UID: \"3e96ce5a1ede90039e45a1c264277675\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.502556 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.502443    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e96ce5a1ede90039e45a1c264277675-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal\" (UID: \"3e96ce5a1ede90039e45a1c264277675\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.517425 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.517408    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.618165 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.618139    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.670324 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.670293    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.674112 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:18.674094    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal"
May 26 11:27:18.718882 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.718723    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.819234 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.819166    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:18.919673 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:18.919643    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:19.009153 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.009129    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 11:27:19.009706 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.009261    2563 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:27:19.020283 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:19.020260    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:19.099966 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.099877    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 11:27:19.114944 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.114919    2563 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:27:19.119281 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.119248    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 11:22:18 +0000 UTC" deadline="2028-01-01 09:17:52.673703647 +0000 UTC"
May 26 11:27:19.119281 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.119280    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14037h50m33.55442609s"
May 26 11:27:19.120355 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:19.120333    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:19.136890 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.136860    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-29kj6"
May 26 11:27:19.144477 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.144454    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-29kj6"
May 26 11:27:19.208879 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:19.208849    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e96ce5a1ede90039e45a1c264277675.slice/crio-0a0d671eb9252f6e2c37f09eef55367baf0b1de8ba17eb7c44dd3670b673c172 WatchSource:0}: Error finding container 0a0d671eb9252f6e2c37f09eef55367baf0b1de8ba17eb7c44dd3670b673c172: Status 404 returned error can't find the container with id 0a0d671eb9252f6e2c37f09eef55367baf0b1de8ba17eb7c44dd3670b673c172
May 26 11:27:19.209078 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:19.209058    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6decc4be96339c77a5112ef19602d9b6.slice/crio-5a842ff322ebf959cf752f3a5e619497184affcb64623ec860c8054fbf8ed233 WatchSource:0}: Error finding container 5a842ff322ebf959cf752f3a5e619497184affcb64623ec860c8054fbf8ed233: Status 404 returned error can't find the container with id 5a842ff322ebf959cf752f3a5e619497184affcb64623ec860c8054fbf8ed233
May 26 11:27:19.213585 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.213570    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:27:19.221275 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:19.221255    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:19.240727 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.240683    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal" event={"ID":"6decc4be96339c77a5112ef19602d9b6","Type":"ContainerStarted","Data":"5a842ff322ebf959cf752f3a5e619497184affcb64623ec860c8054fbf8ed233"}
May 26 11:27:19.241546 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.241528    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal" event={"ID":"3e96ce5a1ede90039e45a1c264277675","Type":"ContainerStarted","Data":"0a0d671eb9252f6e2c37f09eef55367baf0b1de8ba17eb7c44dd3670b673c172"}
May 26 11:27:19.317686 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.317663    2563 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:27:19.322000 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:19.321985    2563 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-138-207.ec2.internal\" not found"
May 26 11:27:19.375931 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.375865    2563 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:27:19.400175 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.400144    2563 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal"
May 26 11:27:19.412928 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.412907    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:27:19.415067 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.415053    2563 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal"
May 26 11:27:19.421168 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.421151    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:27:19.541481 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:19.541454    2563 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:27:20.038592 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.038561    2563 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:27:20.079944 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.079919    2563 apiserver.go:52] "Watching apiserver"
May 26 11:27:20.090005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.089977    2563 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 11:27:20.091977 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.091944    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq","openshift-dns/node-resolver-6t2bn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal","openshift-multus/multus-fbpkr","openshift-network-diagnostics/network-check-target-5tc55","openshift-network-operator/iptables-alerter-lx59v","openshift-ovn-kubernetes/ovnkube-node-w89sb","kube-system/konnectivity-agent-bh49d","kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fl6mp","openshift-image-registry/node-ca-4bwpv","openshift-multus/multus-additional-cni-plugins-k9bkq","openshift-multus/network-metrics-daemon-sp6l5"]
May 26 11:27:20.094627 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.094602    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.096688 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.096653    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 11:27:20.096989 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.096972    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 11:27:20.097368 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.097351    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 11:27:20.097730 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.097706    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5vm4s\""
May 26 11:27:20.098384 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.098364    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 11:27:20.098628 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.098611    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 11:27:20.098887 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.098245    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 11:27:20.100461 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.100444    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.100564 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.100539    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.102057 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.102040    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mc86r\""
May 26 11:27:20.102392 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.102229    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 11:27:20.102717 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.102695    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.102830 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.102807    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 11:27:20.104227 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.103094    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 11:27:20.104227 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.103257    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cv2dv\""
May 26 11:27:20.104227 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.103402    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 11:27:20.104227 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.103582    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 11:27:20.104227 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.103730    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 11:27:20.104227 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.104131    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 11:27:20.105701 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.105419    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:20.105701 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.105519    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:20.106296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.105934    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 11:27:20.106296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.105987    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 11:27:20.106296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.106090    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ctmvl\""
May 26 11:27:20.107930 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.107912    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:20.109299 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.109283    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 11:27:20.109945 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.109813    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 11:27:20.109945 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.109823    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5jcx7\""
May 26 11:27:20.110193 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110158    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.110559 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-run-netns\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.110641 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.110641 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-ovn\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.110641 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-log-socket\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.110811 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-cni-netd\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.110811 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-system-cni-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.110811 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-cni-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.110811 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-node-log\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.110811 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-conf-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.110811 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-multus-certs\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.110811 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-etc-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovnkube-config\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c260307a-5e7d-484e-ad49-0844501c96e1-cni-binary-copy\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-k8s-cni-cncf-io\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4wg\" (UniqueName: \"kubernetes.io/projected/c260307a-5e7d-484e-ad49-0844501c96e1-kube-api-access-nb4wg\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.110990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-kubelet\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-netns\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-hostroot\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-systemd\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-var-lib-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-env-overrides\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8eb48a6b-da7a-4dc2-963e-532b398e5974-hosts-file\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-slash\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-cni-multus\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-systemd-units\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-cni-bin\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4fjv\" (UniqueName: \"kubernetes.io/projected/8bab4db1-10bc-4fbf-945e-e0e88fca597b-kube-api-access-r4fjv\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj77f\" (UniqueName: \"kubernetes.io/projected/8eb48a6b-da7a-4dc2-963e-532b398e5974-kube-api-access-jj77f\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-cnibin\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-os-release\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-socket-dir-parent\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-cni-bin\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.111508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovn-node-metrics-cert\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.112257 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovnkube-script-lib\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.112257 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb48a6b-da7a-4dc2-963e-532b398e5974-tmp-dir\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.112257 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-kubelet\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.112257 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c260307a-5e7d-484e-ad49-0844501c96e1-multus-daemon-config\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.112257 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.111625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-etc-kubernetes\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.112473 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.112400    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.113358 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.112828    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 11:27:20.113358 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.113049    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 11:27:20.113358 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.113222    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 11:27:20.113530 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.113402    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t4h6s\""
May 26 11:27:20.114287 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.113929    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 11:27:20.114287 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.113983    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 11:27:20.114287 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.114149    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-66h8r\""
May 26 11:27:20.114999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.114539    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.114999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.114648    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.116679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.116116    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 11:27:20.116679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.116284    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dprfz\""
May 26 11:27:20.116679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.116384    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 11:27:20.116679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.116461    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 11:27:20.116679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.116549    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-clmmv\""
May 26 11:27:20.116679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.116604    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 11:27:20.116679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.116607    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 11:27:20.117019 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.116786    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:20.117019 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.116859    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:20.145048 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.145018    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:22:19 +0000 UTC" deadline="2027-12-16 08:07:19.121151248 +0000 UTC"
May 26 11:27:20.145162 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.145048    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13652h39m58.976106695s"
May 26 11:27:20.200751 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.200721    2563 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 11:27:20.211774 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-kubernetes\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.211892 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211780    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-slash\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.211892 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211823    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.211892 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-host-slash\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.211892 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht27t\" (UniqueName: \"kubernetes.io/projected/491900cc-f931-4534-b4fd-65d25e7fae85-kube-api-access-ht27t\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.211892 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc9937cb-665a-4714-b717-b577cedcd50f-konnectivity-ca\") pod \"konnectivity-agent-bh49d\" (UID: \"bc9937cb-665a-4714-b717-b577cedcd50f\") " pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-cnibin\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5v4\" (UniqueName: \"kubernetes.io/projected/2a46dace-93be-4609-85f5-1682943deff4-kube-api-access-8t5v4\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211967    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-slash\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.211977    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-cni-bin\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212015    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jj77f\" (UniqueName: \"kubernetes.io/projected/8eb48a6b-da7a-4dc2-963e-532b398e5974-kube-api-access-jj77f\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212046    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-cnibin\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212055    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.212176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212111    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-cni-bin\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212181    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-os-release\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212198    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-socket-dir-parent\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212242    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-cni-bin\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212263    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-kubelet\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212325    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-cnibin\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212335    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-kubelet\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/132f1313-f019-435a-98ca-62608bdbbecd-host\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212378    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212374    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-socket-dir-parent\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212413    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovnkube-script-lib\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212423    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212461    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb48a6b-da7a-4dc2-963e-532b398e5974-tmp-dir\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.212538 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212535    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-os-release\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnmm\" (UniqueName: \"kubernetes.io/projected/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-kube-api-access-tdnmm\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-var-lib-kubelet\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcl2f\" (UniqueName: \"kubernetes.io/projected/132f1313-f019-435a-98ca-62608bdbbecd-kube-api-access-vcl2f\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-cni-binary-copy\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212716    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-run-netns\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212738    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb48a6b-da7a-4dc2-963e-532b398e5974-tmp-dir\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212776    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-log-socket\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212828    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-cni-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212842    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-cni-bin\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212875    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-run-netns\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212905    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-cni-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212963    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-log-socket\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.212993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xmv\" (UniqueName: \"kubernetes.io/projected/7712945a-7f82-4e15-8b15-38ac03441d91-kube-api-access-n2xmv\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213013    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-multus-certs\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.213156 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-sys\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213102    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovnkube-config\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysctl-d\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-host\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213217    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-kubelet\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213237    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-netns\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213251    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-hostroot\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-device-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213340    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-systemd\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213357    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-env-overrides\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-run\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc7361ff-06fe-4836-b2f3-40ee04615ebb-tmp\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213469    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-cni-multus\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-lib-modules\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213492    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovnkube-script-lib\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfb4z\" (UniqueName: \"kubernetes.io/projected/cc7361ff-06fe-4836-b2f3-40ee04615ebb-kube-api-access-lfb4z\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.213919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-os-release\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213609    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-systemd-units\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213625    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r4fjv\" (UniqueName: \"kubernetes.io/projected/8bab4db1-10bc-4fbf-945e-e0e88fca597b-kube-api-access-r4fjv\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213653    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c260307a-5e7d-484e-ad49-0844501c96e1-multus-daemon-config\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213665    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-hostroot\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-iptables-alerter-script\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-sys-fs\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-systemd\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-modprobe-d\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213865    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovn-node-metrics-cert\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213893    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-etc-kubernetes\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213912    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213927    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-ovn\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213942    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-cni-netd\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213963    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-system-cni-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.213986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysconfig\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysctl-conf\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.214661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-system-cni-dir\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214029    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-env-overrides\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214052    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-node-log\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214072    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-conf-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214080    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-multus-certs\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214087    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-tuned\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214156    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-etc-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214183    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c260307a-5e7d-484e-ad49-0844501c96e1-cni-binary-copy\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214202    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-k8s-cni-cncf-io\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214231    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4wg\" (UniqueName: \"kubernetes.io/projected/c260307a-5e7d-484e-ad49-0844501c96e1-kube-api-access-nb4wg\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-socket-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc9937cb-665a-4714-b717-b577cedcd50f-agent-certs\") pod \"konnectivity-agent-bh49d\" (UID: \"bc9937cb-665a-4714-b717-b577cedcd50f\") " pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/132f1313-f019-435a-98ca-62608bdbbecd-serviceca\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214466    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-var-lib-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.215407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214480    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovnkube-config\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214519    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8eb48a6b-da7a-4dc2-963e-532b398e5974-hosts-file\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214526    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-var-lib-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-registration-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214578    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8eb48a6b-da7a-4dc2-963e-532b398e5974-hosts-file\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214618    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-kubelet\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214652    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-systemd\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214652    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-netns\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214695    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-var-lib-cni-multus\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.214729    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-systemd-units\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215315    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c260307a-5e7d-484e-ad49-0844501c96e1-multus-daemon-config\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215344    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-node-log\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215367    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-multus-conf-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215488    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215600    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-etc-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215886    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-host-cni-netd\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215928    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-etc-kubernetes\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215965    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c260307a-5e7d-484e-ad49-0844501c96e1-cni-binary-copy\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.216130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.215969    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-openvswitch\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216870 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.216016    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bab4db1-10bc-4fbf-945e-e0e88fca597b-run-ovn\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.216870 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.216049    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-host-run-k8s-cni-cncf-io\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.216870 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.216065    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c260307a-5e7d-484e-ad49-0844501c96e1-system-cni-dir\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.219150 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.219128    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj77f\" (UniqueName: \"kubernetes.io/projected/8eb48a6b-da7a-4dc2-963e-532b398e5974-kube-api-access-jj77f\") pod \"node-resolver-6t2bn\" (UID: \"8eb48a6b-da7a-4dc2-963e-532b398e5974\") " pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.219377 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.219360    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bab4db1-10bc-4fbf-945e-e0e88fca597b-ovn-node-metrics-cert\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.219568 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.219552    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:27:20.219568 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.219570    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:27:20.219698 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.219580    2563 projected.go:194] Error preparing data for projected volume kube-api-access-jvtv2 for pod openshift-network-diagnostics/network-check-target-5tc55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:20.219698 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.219645    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2 podName:f4d81d1d-211d-4896-b893-c8e472dbece8 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:20.719611061 +0000 UTC m=+3.082560707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jvtv2" (UniqueName: "kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2") pod "network-check-target-5tc55" (UID: "f4d81d1d-211d-4896-b893-c8e472dbece8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:20.221211 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.221192    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4fjv\" (UniqueName: \"kubernetes.io/projected/8bab4db1-10bc-4fbf-945e-e0e88fca597b-kube-api-access-r4fjv\") pod \"ovnkube-node-w89sb\" (UID: \"8bab4db1-10bc-4fbf-945e-e0e88fca597b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.225750 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.225732    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4wg\" (UniqueName: \"kubernetes.io/projected/c260307a-5e7d-484e-ad49-0844501c96e1-kube-api-access-nb4wg\") pod \"multus-fbpkr\" (UID: \"c260307a-5e7d-484e-ad49-0844501c96e1\") " pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.315354 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315270    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.315354 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315324    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-tuned\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.315354 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315352    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-socket-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.315354 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315367    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc9937cb-665a-4714-b717-b577cedcd50f-agent-certs\") pod \"konnectivity-agent-bh49d\" (UID: \"bc9937cb-665a-4714-b717-b577cedcd50f\") " pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315391    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/132f1313-f019-435a-98ca-62608bdbbecd-serviceca\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315416    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315443    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-registration-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315467    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-kubernetes\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315495    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-host-slash\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315517    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ht27t\" (UniqueName: \"kubernetes.io/projected/491900cc-f931-4534-b4fd-65d25e7fae85-kube-api-access-ht27t\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315533    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc9937cb-665a-4714-b717-b577cedcd50f-konnectivity-ca\") pod \"konnectivity-agent-bh49d\" (UID: \"bc9937cb-665a-4714-b717-b577cedcd50f\") " pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315554    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-cnibin\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315576    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315605    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5v4\" (UniqueName: \"kubernetes.io/projected/2a46dace-93be-4609-85f5-1682943deff4-kube-api-access-8t5v4\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315625    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/132f1313-f019-435a-98ca-62608bdbbecd-host\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.315649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315652    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnmm\" (UniqueName: \"kubernetes.io/projected/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-kube-api-access-tdnmm\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315679    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-var-lib-kubelet\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315694    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vcl2f\" (UniqueName: \"kubernetes.io/projected/132f1313-f019-435a-98ca-62608bdbbecd-kube-api-access-vcl2f\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315712    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-cni-binary-copy\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315737    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315755    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315770    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xmv\" (UniqueName: \"kubernetes.io/projected/7712945a-7f82-4e15-8b15-38ac03441d91-kube-api-access-n2xmv\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315808    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-sys\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315824    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysctl-d\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315844    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-host\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315867    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-device-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315887    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315911    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-run\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315933    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc7361ff-06fe-4836-b2f3-40ee04615ebb-tmp\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.315964    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-lib-modules\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316007    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfb4z\" (UniqueName: \"kubernetes.io/projected/cc7361ff-06fe-4836-b2f3-40ee04615ebb-kube-api-access-lfb4z\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316029    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-os-release\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.316186 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316052    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-iptables-alerter-script\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316087    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-sys-fs\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316104    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-systemd\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316122    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-modprobe-d\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316140    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysconfig\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316157    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysctl-conf\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316187    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-system-cni-dir\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316280    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-system-cni-dir\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316349    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316760    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.316854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.316830    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-cnibin\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.317261 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.317154    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:20.317261 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.317220    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:20.817201968 +0000 UTC m=+3.180151614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:20.317336 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317298    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc9937cb-665a-4714-b717-b577cedcd50f-konnectivity-ca\") pod \"konnectivity-agent-bh49d\" (UID: \"bc9937cb-665a-4714-b717-b577cedcd50f\") " pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:20.317399 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317385    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-socket-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.317466 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317452    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-sys\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.317540 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317518    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.317574 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317546    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysctl-d\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.317614 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317577    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-host\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.317614 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317603    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-device-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.317712 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317699    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.317743 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317734    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-run\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317835    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/132f1313-f019-435a-98ca-62608bdbbecd-host\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317904    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-kubernetes\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317917    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-systemd\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317952    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-sys-fs\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.317989    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysctl-conf\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318152    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-modprobe-d\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318169    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-lib-modules\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318182    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/491900cc-f931-4534-b4fd-65d25e7fae85-registration-dir\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318213    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-var-lib-kubelet\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318256    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-sysconfig\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318304    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a46dace-93be-4609-85f5-1682943deff4-os-release\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318542    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318619    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/132f1313-f019-435a-98ca-62608bdbbecd-serviceca\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318695    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a46dace-93be-4609-85f5-1682943deff4-cni-binary-copy\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318833    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-host-slash\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.318888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.318838    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-iptables-alerter-script\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.319662 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.319007    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc7361ff-06fe-4836-b2f3-40ee04615ebb-etc-tuned\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.320331 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.320303    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc9937cb-665a-4714-b717-b577cedcd50f-agent-certs\") pod \"konnectivity-agent-bh49d\" (UID: \"bc9937cb-665a-4714-b717-b577cedcd50f\") " pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:20.325763 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.325723    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcl2f\" (UniqueName: \"kubernetes.io/projected/132f1313-f019-435a-98ca-62608bdbbecd-kube-api-access-vcl2f\") pod \"node-ca-4bwpv\" (UID: \"132f1313-f019-435a-98ca-62608bdbbecd\") " pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.326687 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.326661    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5v4\" (UniqueName: \"kubernetes.io/projected/2a46dace-93be-4609-85f5-1682943deff4-kube-api-access-8t5v4\") pod \"multus-additional-cni-plugins-k9bkq\" (UID: \"2a46dace-93be-4609-85f5-1682943deff4\") " pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.327090 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.327068    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xmv\" (UniqueName: \"kubernetes.io/projected/7712945a-7f82-4e15-8b15-38ac03441d91-kube-api-access-n2xmv\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:20.327334 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.327296    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc7361ff-06fe-4836-b2f3-40ee04615ebb-tmp\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.327422 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.327363    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfb4z\" (UniqueName: \"kubernetes.io/projected/cc7361ff-06fe-4836-b2f3-40ee04615ebb-kube-api-access-lfb4z\") pod \"tuned-fl6mp\" (UID: \"cc7361ff-06fe-4836-b2f3-40ee04615ebb\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.328304 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.328283    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnmm\" (UniqueName: \"kubernetes.io/projected/3dcca82f-ff3d-4fe9-adeb-ca3b84741b77-kube-api-access-tdnmm\") pod \"iptables-alerter-lx59v\" (UID: \"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77\") " pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.328693 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.328671    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht27t\" (UniqueName: \"kubernetes.io/projected/491900cc-f931-4534-b4fd-65d25e7fae85-kube-api-access-ht27t\") pod \"aws-ebs-csi-driver-node-2lkcq\" (UID: \"491900cc-f931-4534-b4fd-65d25e7fae85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.410305 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.410269    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:20.417059 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.417040    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6t2bn"
May 26 11:27:20.431761 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.431734    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbpkr"
May 26 11:27:20.437431 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.437401    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lx59v"
May 26 11:27:20.444308 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.444288    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:20.451871 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.451852    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq"
May 26 11:27:20.458461 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.458443    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fl6mp"
May 26 11:27:20.463966 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.463942    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4bwpv"
May 26 11:27:20.468465 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.468447    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k9bkq"
May 26 11:27:20.720178 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.720095    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:20.720380 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.720246    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:27:20.720380 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.720267    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:27:20.720380 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.720280    2563 projected.go:194] Error preparing data for projected volume kube-api-access-jvtv2 for pod openshift-network-diagnostics/network-check-target-5tc55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:20.720380 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.720350    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2 podName:f4d81d1d-211d-4896-b893-c8e472dbece8 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:21.720331948 +0000 UTC m=+4.083281596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvtv2" (UniqueName: "kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2") pod "network-check-target-5tc55" (UID: "f4d81d1d-211d-4896-b893-c8e472dbece8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:20.820710 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:20.820679    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:20.820886 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.820851    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:20.820936 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:20.820916    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:21.820902155 +0000 UTC m=+4.183851802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:20.845668 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.845637    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a46dace_93be_4609_85f5_1682943deff4.slice/crio-c81dedcc1418185c689f161aeb278070041219e3e4a8bf0c38c36c85275cbe02 WatchSource:0}: Error finding container c81dedcc1418185c689f161aeb278070041219e3e4a8bf0c38c36c85275cbe02: Status 404 returned error can't find the container with id c81dedcc1418185c689f161aeb278070041219e3e4a8bf0c38c36c85275cbe02
May 26 11:27:20.846451 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.846427    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132f1313_f019_435a_98ca_62608bdbbecd.slice/crio-2f61bf990c8ad93c427d6e8e72b52aa8467ffc187540abe833b2924534af0e45 WatchSource:0}: Error finding container 2f61bf990c8ad93c427d6e8e72b52aa8467ffc187540abe833b2924534af0e45: Status 404 returned error can't find the container with id 2f61bf990c8ad93c427d6e8e72b52aa8467ffc187540abe833b2924534af0e45
May 26 11:27:20.847145 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.847103    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bab4db1_10bc_4fbf_945e_e0e88fca597b.slice/crio-066cdff195c00349847c0b899d08370a1d9a580c3e1e24cda1524273ef4c4eb0 WatchSource:0}: Error finding container 066cdff195c00349847c0b899d08370a1d9a580c3e1e24cda1524273ef4c4eb0: Status 404 returned error can't find the container with id 066cdff195c00349847c0b899d08370a1d9a580c3e1e24cda1524273ef4c4eb0
May 26 11:27:20.847825 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.847750    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc260307a_5e7d_484e_ad49_0844501c96e1.slice/crio-b71446e4061f30ce621983199452d7d161c1c16152395e01a729a9e713513aad WatchSource:0}: Error finding container b71446e4061f30ce621983199452d7d161c1c16152395e01a729a9e713513aad: Status 404 returned error can't find the container with id b71446e4061f30ce621983199452d7d161c1c16152395e01a729a9e713513aad
May 26 11:27:20.848724 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.848662    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9937cb_665a_4714_b717_b577cedcd50f.slice/crio-76fca1a3d273d148b8f57284aa6223c0a135e5f2f8d11ec0123a9973425340b3 WatchSource:0}: Error finding container 76fca1a3d273d148b8f57284aa6223c0a135e5f2f8d11ec0123a9973425340b3: Status 404 returned error can't find the container with id 76fca1a3d273d148b8f57284aa6223c0a135e5f2f8d11ec0123a9973425340b3
May 26 11:27:20.871129 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.871099    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491900cc_f931_4534_b4fd_65d25e7fae85.slice/crio-9cf3cf541e364fc8e07ab91153e64e5dc5d5292a96f519963d4b2149a7085b41 WatchSource:0}: Error finding container 9cf3cf541e364fc8e07ab91153e64e5dc5d5292a96f519963d4b2149a7085b41: Status 404 returned error can't find the container with id 9cf3cf541e364fc8e07ab91153e64e5dc5d5292a96f519963d4b2149a7085b41
May 26 11:27:20.871711 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.871689    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcca82f_ff3d_4fe9_adeb_ca3b84741b77.slice/crio-8ccebc32b805337ac0f180613994fd549e14d60a3573aaf5be8833ae389fe6a6 WatchSource:0}: Error finding container 8ccebc32b805337ac0f180613994fd549e14d60a3573aaf5be8833ae389fe6a6: Status 404 returned error can't find the container with id 8ccebc32b805337ac0f180613994fd549e14d60a3573aaf5be8833ae389fe6a6
May 26 11:27:20.872644 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.872622    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7361ff_06fe_4836_b2f3_40ee04615ebb.slice/crio-dd989704277db0de60a9c9570c97778dc726850f977db6e7de38101c4f20222c WatchSource:0}: Error finding container dd989704277db0de60a9c9570c97778dc726850f977db6e7de38101c4f20222c: Status 404 returned error can't find the container with id dd989704277db0de60a9c9570c97778dc726850f977db6e7de38101c4f20222c
May 26 11:27:20.874178 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:20.874162    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb48a6b_da7a_4dc2_963e_532b398e5974.slice/crio-2dd29fcd3c8efc45fe0b8a74cf5e8de65b1cad857a555a288510a415415046f7 WatchSource:0}: Error finding container 2dd29fcd3c8efc45fe0b8a74cf5e8de65b1cad857a555a288510a415415046f7: Status 404 returned error can't find the container with id 2dd29fcd3c8efc45fe0b8a74cf5e8de65b1cad857a555a288510a415415046f7
May 26 11:27:21.145584 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.145403    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:22:19 +0000 UTC" deadline="2028-02-25 12:40:01.938704694 +0000 UTC"
May 26 11:27:21.145584 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.145585    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15361h12m40.793123211s"
May 26 11:27:21.245666 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.245638    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal" event={"ID":"6decc4be96339c77a5112ef19602d9b6","Type":"ContainerStarted","Data":"684c4be11d0612ba3f92a3cf899c6bccb38365d8a85c37a531195584846f31b4"}
May 26 11:27:21.246811 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.246767    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6t2bn" event={"ID":"8eb48a6b-da7a-4dc2-963e-532b398e5974","Type":"ContainerStarted","Data":"2dd29fcd3c8efc45fe0b8a74cf5e8de65b1cad857a555a288510a415415046f7"}
May 26 11:27:21.247749 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.247715    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fl6mp" event={"ID":"cc7361ff-06fe-4836-b2f3-40ee04615ebb","Type":"ContainerStarted","Data":"dd989704277db0de60a9c9570c97778dc726850f977db6e7de38101c4f20222c"}
May 26 11:27:21.248933 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.248910    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq" event={"ID":"491900cc-f931-4534-b4fd-65d25e7fae85","Type":"ContainerStarted","Data":"9cf3cf541e364fc8e07ab91153e64e5dc5d5292a96f519963d4b2149a7085b41"}
May 26 11:27:21.250132 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.250109    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bh49d" event={"ID":"bc9937cb-665a-4714-b717-b577cedcd50f","Type":"ContainerStarted","Data":"76fca1a3d273d148b8f57284aa6223c0a135e5f2f8d11ec0123a9973425340b3"}
May 26 11:27:21.251085 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.251070    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4bwpv" event={"ID":"132f1313-f019-435a-98ca-62608bdbbecd","Type":"ContainerStarted","Data":"2f61bf990c8ad93c427d6e8e72b52aa8467ffc187540abe833b2924534af0e45"}
May 26 11:27:21.252194 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.252163    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerStarted","Data":"c81dedcc1418185c689f161aeb278070041219e3e4a8bf0c38c36c85275cbe02"}
May 26 11:27:21.253628 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.253606    2563 generic.go:358] "Generic (PLEG): container finished" podID="3e96ce5a1ede90039e45a1c264277675" containerID="85a1be75ca0ca5e3d298f04289bac3fd80446b6119fc77d8c3b503b2c1516a8a" exitCode=0
May 26 11:27:21.253712 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.253675    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal" event={"ID":"3e96ce5a1ede90039e45a1c264277675","Type":"ContainerDied","Data":"85a1be75ca0ca5e3d298f04289bac3fd80446b6119fc77d8c3b503b2c1516a8a"}
May 26 11:27:21.256737 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.256716    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lx59v" event={"ID":"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77","Type":"ContainerStarted","Data":"8ccebc32b805337ac0f180613994fd549e14d60a3573aaf5be8833ae389fe6a6"}
May 26 11:27:21.257417 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.257373    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-207.ec2.internal" podStartSLOduration=2.257339706 podStartE2EDuration="2.257339706s" podCreationTimestamp="2026-05-26 11:27:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:27:21.256465845 +0000 UTC m=+3.619415510" watchObservedRunningTime="2026-05-26 11:27:21.257339706 +0000 UTC m=+3.620289377"
May 26 11:27:21.258214 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.258195    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbpkr" event={"ID":"c260307a-5e7d-484e-ad49-0844501c96e1","Type":"ContainerStarted","Data":"b71446e4061f30ce621983199452d7d161c1c16152395e01a729a9e713513aad"}
May 26 11:27:21.259250 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.259230    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"066cdff195c00349847c0b899d08370a1d9a580c3e1e24cda1524273ef4c4eb0"}
May 26 11:27:21.728083 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.727345    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:21.728083 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:21.727523    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:27:21.728083 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:21.727541    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:27:21.728083 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:21.727553    2563 projected.go:194] Error preparing data for projected volume kube-api-access-jvtv2 for pod openshift-network-diagnostics/network-check-target-5tc55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:21.728083 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:21.727610    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2 podName:f4d81d1d-211d-4896-b893-c8e472dbece8 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:23.727591426 +0000 UTC m=+6.090541116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvtv2" (UniqueName: "kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2") pod "network-check-target-5tc55" (UID: "f4d81d1d-211d-4896-b893-c8e472dbece8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:21.827965 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:21.827918    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:21.828164 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:21.828125    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:21.828231 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:21.828187    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:23.828169527 +0000 UTC m=+6.191119190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:22.250948 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:22.250917    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:22.251406 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:22.251112    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:22.255081 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:22.255057    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:22.255250 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:22.255226    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:22.278716 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:22.278666    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal" event={"ID":"3e96ce5a1ede90039e45a1c264277675","Type":"ContainerStarted","Data":"1474b428a71188fe7532c8a84f4dec56e8b5b423b9310009293632abb7423549"}
May 26 11:27:22.290460 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:22.290332    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-207.ec2.internal" podStartSLOduration=3.29031742 podStartE2EDuration="3.29031742s" podCreationTimestamp="2026-05-26 11:27:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:27:22.289272759 +0000 UTC m=+4.652222430" watchObservedRunningTime="2026-05-26 11:27:22.29031742 +0000 UTC m=+4.653267088"
May 26 11:27:23.245282 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.245196    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-99q8l"]
May 26 11:27:23.248142 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.248110    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.248261 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.248192    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:23.341778 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.341692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.341778 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.341742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-dbus\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.342382 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.341877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-kubelet-config\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.443589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.442814    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-kubelet-config\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.443589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.442897    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.443589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.442926    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-dbus\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.443589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.443170    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-dbus\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.443589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.443232    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-kubelet-config\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.443589 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.443324    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:27:23.443589 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.443378    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret podName:b8f27af5-0b34-4fe2-ad62-afca8eb49a7c nodeName:}" failed. No retries permitted until 2026-05-26 11:27:23.943360107 +0000 UTC m=+6.306309767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret") pod "global-pull-secret-syncer-99q8l" (UID: "b8f27af5-0b34-4fe2-ad62-afca8eb49a7c") : object "kube-system"/"original-pull-secret" not registered
May 26 11:27:23.745647 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.745613    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:23.745891 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.745866    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:27:23.745891 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.745891    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:27:23.746018 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.745903    2563 projected.go:194] Error preparing data for projected volume kube-api-access-jvtv2 for pod openshift-network-diagnostics/network-check-target-5tc55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:23.746018 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.745961    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2 podName:f4d81d1d-211d-4896-b893-c8e472dbece8 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:27.745944074 +0000 UTC m=+10.108893723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvtv2" (UniqueName: "kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2") pod "network-check-target-5tc55" (UID: "f4d81d1d-211d-4896-b893-c8e472dbece8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:23.846861 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.846824    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:23.847024 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.846971    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:23.847086 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.847032    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:27.847017391 +0000 UTC m=+10.209967041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:23.947769 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:23.947731    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:23.947940 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.947925    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:27:23.948034 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:23.948026    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret podName:b8f27af5-0b34-4fe2-ad62-afca8eb49a7c nodeName:}" failed. No retries permitted until 2026-05-26 11:27:24.947988963 +0000 UTC m=+7.310938627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret") pod "global-pull-secret-syncer-99q8l" (UID: "b8f27af5-0b34-4fe2-ad62-afca8eb49a7c") : object "kube-system"/"original-pull-secret" not registered
May 26 11:27:24.241717 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:24.240988    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:24.241717 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:24.241123    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:24.241717 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:24.241516    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:24.241717 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:24.241613    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:24.958606 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:24.958122    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:24.958606 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:24.958251    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:27:24.958606 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:24.958304    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret podName:b8f27af5-0b34-4fe2-ad62-afca8eb49a7c nodeName:}" failed. No retries permitted until 2026-05-26 11:27:26.958291338 +0000 UTC m=+9.321240989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret") pod "global-pull-secret-syncer-99q8l" (UID: "b8f27af5-0b34-4fe2-ad62-afca8eb49a7c") : object "kube-system"/"original-pull-secret" not registered
May 26 11:27:25.238690 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:25.238615    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:25.238877 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:25.238748    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:26.241765 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:26.241737    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:26.242210 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:26.241880    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:26.242303 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:26.242244    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:26.242357 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:26.242334    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:26.976179 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:26.976072    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:26.976348 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:26.976236    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:27:26.976348 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:26.976316    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret podName:b8f27af5-0b34-4fe2-ad62-afca8eb49a7c nodeName:}" failed. No retries permitted until 2026-05-26 11:27:30.976292891 +0000 UTC m=+13.339242543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret") pod "global-pull-secret-syncer-99q8l" (UID: "b8f27af5-0b34-4fe2-ad62-afca8eb49a7c") : object "kube-system"/"original-pull-secret" not registered
May 26 11:27:27.238318 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:27.238235    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:27.238479 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:27.238380    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:27.783097 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:27.782439    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:27.783097 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:27.782681    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:27:27.783097 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:27.782699    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:27:27.783097 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:27.782709    2563 projected.go:194] Error preparing data for projected volume kube-api-access-jvtv2 for pod openshift-network-diagnostics/network-check-target-5tc55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:27.783097 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:27.782759    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2 podName:f4d81d1d-211d-4896-b893-c8e472dbece8 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:35.782743031 +0000 UTC m=+18.145692679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvtv2" (UniqueName: "kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2") pod "network-check-target-5tc55" (UID: "f4d81d1d-211d-4896-b893-c8e472dbece8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:27.883853 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:27.883786    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:27.884033 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:27.883995    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:27.884155 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:27.884076    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:35.884057161 +0000 UTC m=+18.247006822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:28.239825 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:28.239727    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:28.240004 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:28.239876    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:28.240313 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:28.240294    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:28.240410 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:28.240390    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:29.238389 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:29.238352    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:29.238923 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:29.238464    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:30.238876 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:30.238835    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:30.239405 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:30.238951    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:30.239405 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:30.239347    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:30.239510 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:30.239448    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:31.008929 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:31.008899    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:31.009095 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:31.009051    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:27:31.009169 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:31.009131    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret podName:b8f27af5-0b34-4fe2-ad62-afca8eb49a7c nodeName:}" failed. No retries permitted until 2026-05-26 11:27:39.009098162 +0000 UTC m=+21.372047809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret") pod "global-pull-secret-syncer-99q8l" (UID: "b8f27af5-0b34-4fe2-ad62-afca8eb49a7c") : object "kube-system"/"original-pull-secret" not registered
May 26 11:27:31.238357 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:31.238277    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:31.238510 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:31.238399    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:32.238879 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:32.238846    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:32.238879 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:32.238863    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:32.239366 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:32.238958    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:32.239366 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:32.239106    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:33.237907 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:33.237870    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:33.238084 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:33.237987    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:34.238712 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:34.238673    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:34.239127 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:34.238680    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:34.239127 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:34.238823    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:34.239127 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:34.238912    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:35.238367 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:35.238329    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:35.238524 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:35.238469    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:35.844874 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:35.844839    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:35.845304 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:35.845009    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:27:35.845304 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:35.845034    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:27:35.845304 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:35.845047    2563 projected.go:194] Error preparing data for projected volume kube-api-access-jvtv2 for pod openshift-network-diagnostics/network-check-target-5tc55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:35.845304 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:35.845103    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2 podName:f4d81d1d-211d-4896-b893-c8e472dbece8 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:51.845084679 +0000 UTC m=+34.208034326 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvtv2" (UniqueName: "kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2") pod "network-check-target-5tc55" (UID: "f4d81d1d-211d-4896-b893-c8e472dbece8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:35.946154 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:35.946115    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:35.946328 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:35.946283    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:35.946386 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:35.946362    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:51.946338775 +0000 UTC m=+34.309288428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:36.238837 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:36.238739    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:36.238983 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:36.238740    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:36.238983 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:36.238896    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:36.238983 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:36.238941    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:37.237920 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:37.237887    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:37.238350 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:37.238007    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:38.239451 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:38.239422    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:38.239855 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:38.239504    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:38.239855 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:38.239575    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:38.239855 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:38.239695    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:39.070259 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.070229    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:39.070397 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:39.070378    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:27:39.070472 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:39.070461    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret podName:b8f27af5-0b34-4fe2-ad62-afca8eb49a7c nodeName:}" failed. No retries permitted until 2026-05-26 11:27:55.07044331 +0000 UTC m=+37.433392963 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret") pod "global-pull-secret-syncer-99q8l" (UID: "b8f27af5-0b34-4fe2-ad62-afca8eb49a7c") : object "kube-system"/"original-pull-secret" not registered
May 26 11:27:39.238217 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.238068    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:39.238313 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:39.238296    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:39.307252 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.307219    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fl6mp" event={"ID":"cc7361ff-06fe-4836-b2f3-40ee04615ebb","Type":"ContainerStarted","Data":"8ed13ac04a9a4578ea81f6a9daf2159e9b7276b76bece0b998d3ab588b12ca6a"}
May 26 11:27:39.308456 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.308424    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq" event={"ID":"491900cc-f931-4534-b4fd-65d25e7fae85","Type":"ContainerStarted","Data":"b8460def09d79e41fcabfef2b33b37c5fd1e4d00c2fd334b1420fa30e4f87a55"}
May 26 11:27:39.310013 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.309969    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bh49d" event={"ID":"bc9937cb-665a-4714-b717-b577cedcd50f","Type":"ContainerStarted","Data":"551da96e083fc0bcd58d29df326cc8ff5ad9b17f6bd1002ebd1bd53ee9c980ca"}
May 26 11:27:39.311482 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.311214    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4bwpv" event={"ID":"132f1313-f019-435a-98ca-62608bdbbecd","Type":"ContainerStarted","Data":"bce39da079a6962b3c09037cad94e67136e064ac7a16bef51a7701b2cb2d71ab"}
May 26 11:27:39.312587 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.312567    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerStarted","Data":"5490fa14644e5fb4f81043507a968e98c22817178d28198a7c50868cdbfb9d20"}
May 26 11:27:39.313706 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.313689    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbpkr" event={"ID":"c260307a-5e7d-484e-ad49-0844501c96e1","Type":"ContainerStarted","Data":"2f90b509c079a4ae7fcc50de4af7d759bc3ef92209c5fd5f549962048a2c6756"}
May 26 11:27:39.314807 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.314776    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"d4adbf0e33e33200ea0ac369969bbcce3a4751d4458dca9f25cb179801ee83d6"}
May 26 11:27:39.315774 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.315752    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6t2bn" event={"ID":"8eb48a6b-da7a-4dc2-963e-532b398e5974","Type":"ContainerStarted","Data":"ae842415c2af02f8429e6a7d48a140cac4e8073e685f6443317dff6d85fa6737"}
May 26 11:27:39.321649 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.321617    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fl6mp" podStartSLOduration=3.267826481 podStartE2EDuration="21.321608225s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.875281411 +0000 UTC m=+3.238231073" lastFinishedPulling="2026-05-26 11:27:38.929063155 +0000 UTC m=+21.292012817" observedRunningTime="2026-05-26 11:27:39.321541299 +0000 UTC m=+21.684490966" watchObservedRunningTime="2026-05-26 11:27:39.321608225 +0000 UTC m=+21.684557892"
May 26 11:27:39.353000 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.352964    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-fbpkr" podStartSLOduration=3.286654718 podStartE2EDuration="21.352953234s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.869079493 +0000 UTC m=+3.232029140" lastFinishedPulling="2026-05-26 11:27:38.93537801 +0000 UTC m=+21.298327656" observedRunningTime="2026-05-26 11:27:39.352856996 +0000 UTC m=+21.715806665" watchObservedRunningTime="2026-05-26 11:27:39.352953234 +0000 UTC m=+21.715902902"
May 26 11:27:39.368300 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.368263    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bh49d" podStartSLOduration=3.363313627 podStartE2EDuration="21.368250543s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.869075951 +0000 UTC m=+3.232025612" lastFinishedPulling="2026-05-26 11:27:38.874012868 +0000 UTC m=+21.236962528" observedRunningTime="2026-05-26 11:27:39.367606016 +0000 UTC m=+21.730555683" watchObservedRunningTime="2026-05-26 11:27:39.368250543 +0000 UTC m=+21.731200212"
May 26 11:27:39.381924 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.381885    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-6t2bn" podStartSLOduration=3.321569149 podStartE2EDuration="21.381871605s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.875712189 +0000 UTC m=+3.238661839" lastFinishedPulling="2026-05-26 11:27:38.936014647 +0000 UTC m=+21.298964295" observedRunningTime="2026-05-26 11:27:39.381777208 +0000 UTC m=+21.744726878" watchObservedRunningTime="2026-05-26 11:27:39.381871605 +0000 UTC m=+21.744821286"
May 26 11:27:39.393727 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:39.393684    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4bwpv" podStartSLOduration=3.347393621 podStartE2EDuration="21.393671596s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.848051616 +0000 UTC m=+3.211001263" lastFinishedPulling="2026-05-26 11:27:38.894329579 +0000 UTC m=+21.257279238" observedRunningTime="2026-05-26 11:27:39.393384607 +0000 UTC m=+21.756334277" watchObservedRunningTime="2026-05-26 11:27:39.393671596 +0000 UTC m=+21.756621263"
May 26 11:27:40.238786 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.238612    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:40.238786 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.238612    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:40.239123 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:40.238892    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:40.239123 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:40.239004    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:40.318607 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.318572    2563 generic.go:358] "Generic (PLEG): container finished" podID="2a46dace-93be-4609-85f5-1682943deff4" containerID="5490fa14644e5fb4f81043507a968e98c22817178d28198a7c50868cdbfb9d20" exitCode=0
May 26 11:27:40.319016 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.318654    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerDied","Data":"5490fa14644e5fb4f81043507a968e98c22817178d28198a7c50868cdbfb9d20"}
May 26 11:27:40.321157 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.321141    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:27:40.321440 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.321421    2563 generic.go:358] "Generic (PLEG): container finished" podID="8bab4db1-10bc-4fbf-945e-e0e88fca597b" containerID="16033f2bdc8cdff697f483f070310773f1c860b3840dddf18569528e95a671eb" exitCode=1
May 26 11:27:40.321508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.321449    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"a4f19f6323f8faeed8afd2cc6bab89042d9844eef1b44b40b37435ecbab63fb5"}
May 26 11:27:40.321508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.321478    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"b4fd25710ed6e602247fca016f167df5213cdfbdc573f97116d44124f8d4b74a"}
May 26 11:27:40.321508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.321492    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"13a704fbdd4467c7c159d9cfc3374d85a88348696564967bdc08115c272b1d1d"}
May 26 11:27:40.321651 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.321518    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"1065a026c892b8d730a978a1ef34e9c2141590ed369e4b354bb13489085ac34d"}
May 26 11:27:40.321651 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.321535    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerDied","Data":"16033f2bdc8cdff697f483f070310773f1c860b3840dddf18569528e95a671eb"}
May 26 11:27:40.678960 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:40.678937    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 11:27:41.196897 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:41.196718    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T11:27:40.678955836Z","UUID":"724b2b80-113a-4796-9c45-73dae165e940","Handler":null,"Name":"","Endpoint":""}
May 26 11:27:41.198934 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:41.198915    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 11:27:41.199069 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:41.198943    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 11:27:41.238173 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:41.238146    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:41.238356 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:41.238268    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:41.325160 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:41.325118    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lx59v" event={"ID":"3dcca82f-ff3d-4fe9-adeb-ca3b84741b77","Type":"ContainerStarted","Data":"e6c18a89d7950061cab126d5283f048f2507773f25e4298a620bcceabd127395"}
May 26 11:27:41.326567 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:41.326543    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq" event={"ID":"491900cc-f931-4534-b4fd-65d25e7fae85","Type":"ContainerStarted","Data":"2a38acf5b39c1e55978acef1b1e1f4ec8919ff4c2636f0a96c12992ef7bac6a6"}
May 26 11:27:41.338866 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:41.338823    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lx59v" podStartSLOduration=5.285498705 podStartE2EDuration="23.338808482s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.875539043 +0000 UTC m=+3.238488690" lastFinishedPulling="2026-05-26 11:27:38.928848817 +0000 UTC m=+21.291798467" observedRunningTime="2026-05-26 11:27:41.338310797 +0000 UTC m=+23.701260464" watchObservedRunningTime="2026-05-26 11:27:41.338808482 +0000 UTC m=+23.701758147"
May 26 11:27:42.238428 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:42.238391    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:42.238610 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:42.238508    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:42.238610 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:42.238568    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:42.238718 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:42.238694    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:42.329469 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:42.329437    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq" event={"ID":"491900cc-f931-4534-b4fd-65d25e7fae85","Type":"ContainerStarted","Data":"86379ba0872c01f56415ecb99f507ea33c943999e830068861d2e34879c56b86"}
May 26 11:27:42.332178 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:42.332155    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:27:42.332932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:42.332911    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"67b3a66f4848b7ebf2eecc4bbb176409c063418be3b86d58e43aefcefa88e3ae"}
May 26 11:27:42.345988 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:42.345941    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lkcq" podStartSLOduration=3.540802663 podStartE2EDuration="24.345926386s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.875257428 +0000 UTC m=+3.238207074" lastFinishedPulling="2026-05-26 11:27:41.680381147 +0000 UTC m=+24.043330797" observedRunningTime="2026-05-26 11:27:42.345657487 +0000 UTC m=+24.708607155" watchObservedRunningTime="2026-05-26 11:27:42.345926386 +0000 UTC m=+24.708876055"
May 26 11:27:42.544386 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:42.544308    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:42.544876 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:42.544854    2563 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:43.238714 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:43.238636    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:43.238874 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:43.238770    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:44.238829 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:44.238743    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:44.239295 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:44.238743    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:44.239295 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:44.238893    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:44.239295 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:44.238961    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:45.238947 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.238738    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:45.239385 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:45.238987    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:45.338777 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.338745    2563 generic.go:358] "Generic (PLEG): container finished" podID="2a46dace-93be-4609-85f5-1682943deff4" containerID="504d10a4d7785601923d65d8cc08456db2fe787ef05ffe94c6f89deac68a6cc6" exitCode=0
May 26 11:27:45.338940 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.338835    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerDied","Data":"504d10a4d7785601923d65d8cc08456db2fe787ef05ffe94c6f89deac68a6cc6"}
May 26 11:27:45.344970 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.344951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:27:45.345253 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.345227    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"e488f6edad6e79b6c5b73f4720e440933ffcf7b8aa197b6efa517e743b592ccf"}
May 26 11:27:45.345462 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.345446    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:45.345559 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.345546    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:45.345701 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.345652    2563 scope.go:117] "RemoveContainer" containerID="16033f2bdc8cdff697f483f070310773f1c860b3840dddf18569528e95a671eb"
May 26 11:27:45.361745 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.361727    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:45.616363 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.616289    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:45.616537 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.616409    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 11:27:45.616770 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:45.616747    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bh49d"
May 26 11:27:46.238424 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.238270    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:46.238565 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:46.238485    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:46.238565 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.238332    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:46.238677 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:46.238624    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:46.350652 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.350588    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:27:46.351034 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.350942    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" event={"ID":"8bab4db1-10bc-4fbf-945e-e0e88fca597b","Type":"ContainerStarted","Data":"7dfb496b52ba89d3c0ab49868417e59a86a093f681b33609fc0a0c7827d89730"}
May 26 11:27:46.351357 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.351336    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:46.353102 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.353067    2563 generic.go:358] "Generic (PLEG): container finished" podID="2a46dace-93be-4609-85f5-1682943deff4" containerID="f63c7e2bad4b2f62c1e53394543c2dbca2c0a59ff67121729e1f1711d8447260" exitCode=0
May 26 11:27:46.353193 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.353128    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerDied","Data":"f63c7e2bad4b2f62c1e53394543c2dbca2c0a59ff67121729e1f1711d8447260"}
May 26 11:27:46.366898 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.366880    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:27:46.376001 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.375934    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb" podStartSLOduration=10.262291966 podStartE2EDuration="28.375922718s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.84941068 +0000 UTC m=+3.212360340" lastFinishedPulling="2026-05-26 11:27:38.963041445 +0000 UTC m=+21.325991092" observedRunningTime="2026-05-26 11:27:46.375094139 +0000 UTC m=+28.738043806" watchObservedRunningTime="2026-05-26 11:27:46.375922718 +0000 UTC m=+28.738872385"
May 26 11:27:46.678216 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.678141    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-99q8l"]
May 26 11:27:46.678352 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.678271    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:46.678402 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:46.678379    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:46.681229 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.681205    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sp6l5"]
May 26 11:27:46.681362 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.681315    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:46.681437 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:46.681416    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:46.681741 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.681724    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5tc55"]
May 26 11:27:46.681843 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:46.681830    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:46.681921 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:46.681904    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:47.357441 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:47.357354    2563 generic.go:358] "Generic (PLEG): container finished" podID="2a46dace-93be-4609-85f5-1682943deff4" containerID="5cf093dcade03bcb17355439a9c5c5975b9b7ebfb7844d11bf03c9752d9e0ed5" exitCode=0
May 26 11:27:47.357895 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:47.357435    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerDied","Data":"5cf093dcade03bcb17355439a9c5c5975b9b7ebfb7844d11bf03c9752d9e0ed5"}
May 26 11:27:48.240885 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:48.239744    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:48.240885 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:48.239889    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:48.240885 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:48.239899    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:48.240885 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:48.240002    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:48.240885 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:48.240051    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:48.240885 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:48.240127    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:50.238405 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:50.238367    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:50.239005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:50.238367    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:50.239005 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:50.238503    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:27:50.239005 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:50.238565    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-99q8l" podUID="b8f27af5-0b34-4fe2-ad62-afca8eb49a7c"
May 26 11:27:50.239005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:50.238382    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:50.239005 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:50.238693    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tc55" podUID="f4d81d1d-211d-4896-b893-c8e472dbece8"
May 26 11:27:51.859305 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.859082    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:51.859809 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:51.859264    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:27:51.859809 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:51.859377    2563 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:27:51.859809 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:51.859391    2563 projected.go:194] Error preparing data for projected volume kube-api-access-jvtv2 for pod openshift-network-diagnostics/network-check-target-5tc55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:51.859809 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:51.859454    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2 podName:f4d81d1d-211d-4896-b893-c8e472dbece8 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:23.859434582 +0000 UTC m=+66.222384231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jvtv2" (UniqueName: "kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2") pod "network-check-target-5tc55" (UID: "f4d81d1d-211d-4896-b893-c8e472dbece8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:27:51.925647 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.925558    2563 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-138-207.ec2.internal" event="NodeReady"
May 26 11:27:51.925787 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.925681    2563 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 11:27:51.960421 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.960388    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:51.960536 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:51.960514    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:51.960590 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:51.960569    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:23.960549836 +0000 UTC m=+66.323499488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:27:51.962320 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.962294    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"]
May 26 11:27:51.993525 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.993496    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"]
May 26 11:27:51.993684 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.993654    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:51.995644 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.995621    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 26 11:27:51.995761 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.995630    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 26 11:27:51.995761 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.995628    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 26 11:27:51.996057 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:51.996037    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 26 11:27:52.007520 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.007499    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cf4b5ff5-csbf4"]
May 26 11:27:52.007637 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.007621    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"
May 26 11:27:52.009999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.009689    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 26 11:27:52.009999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.009692    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zzfng\""
May 26 11:27:52.027542 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.027521    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"]
May 26 11:27:52.027673 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.027654    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.029990 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.029862    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 11:27:52.029990 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.029876    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 11:27:52.029990 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.029932    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 11:27:52.029990 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.029967    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-697bc\""
May 26 11:27:52.030234 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.030204    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"]
May 26 11:27:52.030234 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.030224    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"]
May 26 11:27:52.030330 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.030240    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"]
May 26 11:27:52.030330 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.030254    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf4b5ff5-csbf4"]
May 26 11:27:52.030330 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.030269    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6d6tg"]
May 26 11:27:52.030464 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.030363    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.031831 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.031814    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6d6tg"]
May 26 11:27:52.032048 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.032030    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:52.032214 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.032107    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 26 11:27:52.032467 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.032448    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 26 11:27:52.032562 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.032486    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 26 11:27:52.033086 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.033058    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 26 11:27:52.033834 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.033722    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 11:27:52.034190 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.034173    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 11:27:52.034190 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.034186    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 11:27:52.034934 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.034916    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gmhr4\""
May 26 11:27:52.036485 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.036466    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 11:27:52.083660 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.083633    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-85gbr"]
May 26 11:27:52.085928 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.085908    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.087677 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.087659    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 11:27:52.088003 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.087985    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sqkc6\""
May 26 11:27:52.088092 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.087985    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 11:27:52.095078 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.095059    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-85gbr"]
May 26 11:27:52.162223 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.162426 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/455cc5be-f98e-4661-8935-bc30143a23e1-ca-trust-extracted\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.162426 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-ca\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.162426 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-bound-sa-token\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.162426 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5751358a-1f75-42f5-87e2-e09723f578e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-774f68f574-xpv4f\" (UID: \"5751358a-1f75-42f5-87e2-e09723f578e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"
May 26 11:27:52.162426 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfpg\" (UniqueName: \"kubernetes.io/projected/263e3474-ec60-4909-8953-24916d1eb74a-kube-api-access-kqfpg\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:52.162426 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkq8l\" (UniqueName: \"kubernetes.io/projected/5751358a-1f75-42f5-87e2-e09723f578e7-kube-api-access-dkq8l\") pod \"managed-serviceaccount-addon-agent-774f68f574-xpv4f\" (UID: \"5751358a-1f75-42f5-87e2-e09723f578e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"
May 26 11:27:52.162426 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw64h\" (UniqueName: \"kubernetes.io/projected/13850f0b-ef14-4b33-854b-943840573b6b-kube-api-access-xw64h\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.162426 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-trusted-ca\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-installation-pull-secrets\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-registry-certificates\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13850f0b-ef14-4b33-854b-943840573b6b-tmp\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-hub\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhp8w\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-kube-api-access-zhp8w\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/13850f0b-ef14-4b33-854b-943840573b6b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrls\" (UniqueName: \"kubernetes.io/projected/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-kube-api-access-9nrls\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-image-registry-private-configuration\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.162860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.162819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.238730 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.238647    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:52.238912 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.238647    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:27:52.239125 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.238664    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:27:52.241870 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.241849    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wnncn\""
May 26 11:27:52.243152 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.243133    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jz9wk\""
May 26 11:27:52.243369 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.243355    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:27:52.243625 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.243600    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 11:27:52.243864 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.243850    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:27:52.244116 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.244102    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:27:52.263760 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263741    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.263865 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263771    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.263865 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263831    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.263980 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263864    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/455cc5be-f98e-4661-8935-bc30143a23e1-ca-trust-extracted\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.263980 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263896    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-ca\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.263980 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263916    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-bound-sa-token\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.263980 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d91a50-338c-450a-a121-2b08b9204793-config-volume\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.263980 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.264170 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.263990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zwn\" (UniqueName: \"kubernetes.io/projected/c0d91a50-338c-450a-a121-2b08b9204793-kube-api-access-78zwn\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.264170 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264011    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5751358a-1f75-42f5-87e2-e09723f578e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-774f68f574-xpv4f\" (UID: \"5751358a-1f75-42f5-87e2-e09723f578e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"
May 26 11:27:52.264170 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264038    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfpg\" (UniqueName: \"kubernetes.io/projected/263e3474-ec60-4909-8953-24916d1eb74a-kube-api-access-kqfpg\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:52.264170 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264073    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dkq8l\" (UniqueName: \"kubernetes.io/projected/5751358a-1f75-42f5-87e2-e09723f578e7-kube-api-access-dkq8l\") pod \"managed-serviceaccount-addon-agent-774f68f574-xpv4f\" (UID: \"5751358a-1f75-42f5-87e2-e09723f578e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"
May 26 11:27:52.264170 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264100    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xw64h\" (UniqueName: \"kubernetes.io/projected/13850f0b-ef14-4b33-854b-943840573b6b-kube-api-access-xw64h\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.264170 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264124    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-trusted-ca\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.264170 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264141    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-installation-pull-secrets\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.264170 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264160    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264187    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-registry-certificates\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d91a50-338c-450a-a121-2b08b9204793-tmp-dir\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264236    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13850f0b-ef14-4b33-854b-943840573b6b-tmp\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264258    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-hub\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264290    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zhp8w\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-kube-api-access-zhp8w\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264311    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/13850f0b-ef14-4b33-854b-943840573b6b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264336    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrls\" (UniqueName: \"kubernetes.io/projected/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-kube-api-access-9nrls\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264356    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:52.264450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264382    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-image-registry-private-configuration\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.264779 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.264556    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.264988 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.264960    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:27:52.264988 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.264983    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4b5ff5-csbf4: secret "image-registry-tls" not found
May 26 11:27:52.265151 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.265058    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls podName:455cc5be-f98e-4661-8935-bc30143a23e1 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:52.765040739 +0000 UTC m=+35.127990387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls") pod "image-registry-7cf4b5ff5-csbf4" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1") : secret "image-registry-tls" not found
May 26 11:27:52.265151 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.265126    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/455cc5be-f98e-4661-8935-bc30143a23e1-ca-trust-extracted\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.265450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.265430    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13850f0b-ef14-4b33-854b-943840573b6b-tmp\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.265915 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.265895    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-registry-certificates\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.269874 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.269581    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-image-registry-private-configuration\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.269874 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.269631    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5751358a-1f75-42f5-87e2-e09723f578e7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-774f68f574-xpv4f\" (UID: \"5751358a-1f75-42f5-87e2-e09723f578e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"
May 26 11:27:52.269874 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.269672    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:27:52.269874 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.269720    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-hub\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.269874 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.269732    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert podName:263e3474-ec60-4909-8953-24916d1eb74a nodeName:}" failed. No retries permitted until 2026-05-26 11:27:52.769714459 +0000 UTC m=+35.132664118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert") pod "ingress-canary-6d6tg" (UID: "263e3474-ec60-4909-8953-24916d1eb74a") : secret "canary-serving-cert" not found
May 26 11:27:52.269874 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.269828    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.270205 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.269998    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.270259 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.270232    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/13850f0b-ef14-4b33-854b-943840573b6b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.270591 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.270522    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-trusted-ca\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.271996 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.271975    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-installation-pull-secrets\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.272284 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.272254    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-ca\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.275646 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.275593    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw64h\" (UniqueName: \"kubernetes.io/projected/13850f0b-ef14-4b33-854b-943840573b6b-kube-api-access-xw64h\") pod \"klusterlet-addon-workmgr-7fb9dc6d46-flvgr\" (UID: \"13850f0b-ef14-4b33-854b-943840573b6b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.276009 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.275950    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-bound-sa-token\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.277392 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.277346    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfpg\" (UniqueName: \"kubernetes.io/projected/263e3474-ec60-4909-8953-24916d1eb74a-kube-api-access-kqfpg\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:52.277483 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.277421    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrls\" (UniqueName: \"kubernetes.io/projected/fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11-kube-api-access-9nrls\") pod \"cluster-proxy-proxy-agent-666fd97845-2zbgg\" (UID: \"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.277548 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.277481    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkq8l\" (UniqueName: \"kubernetes.io/projected/5751358a-1f75-42f5-87e2-e09723f578e7-kube-api-access-dkq8l\") pod \"managed-serviceaccount-addon-agent-774f68f574-xpv4f\" (UID: \"5751358a-1f75-42f5-87e2-e09723f578e7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"
May 26 11:27:52.278713 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.278691    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhp8w\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-kube-api-access-zhp8w\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.305585 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.305565    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:27:52.327320 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.327299    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"
May 26 11:27:52.346191 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.346166    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:27:52.366141 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.365171    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d91a50-338c-450a-a121-2b08b9204793-config-volume\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.366141 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.365205    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.366141 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.365225    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78zwn\" (UniqueName: \"kubernetes.io/projected/c0d91a50-338c-450a-a121-2b08b9204793-kube-api-access-78zwn\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.366141 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.365284    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d91a50-338c-450a-a121-2b08b9204793-tmp-dir\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.366141 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.365641    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d91a50-338c-450a-a121-2b08b9204793-tmp-dir\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.366141 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.365941    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:27:52.366141 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.366014    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls podName:c0d91a50-338c-450a-a121-2b08b9204793 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:52.865995456 +0000 UTC m=+35.228945102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls") pod "dns-default-85gbr" (UID: "c0d91a50-338c-450a-a121-2b08b9204793") : secret "dns-default-metrics-tls" not found
May 26 11:27:52.366141 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.366089    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d91a50-338c-450a-a121-2b08b9204793-config-volume\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.374736 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.374714    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zwn\" (UniqueName: \"kubernetes.io/projected/c0d91a50-338c-450a-a121-2b08b9204793-kube-api-access-78zwn\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.768528 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.768490    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:52.768726 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.768657    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:27:52.768726 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.768679    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4b5ff5-csbf4: secret "image-registry-tls" not found
May 26 11:27:52.768871 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.768751    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls podName:455cc5be-f98e-4661-8935-bc30143a23e1 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:53.768729571 +0000 UTC m=+36.131679217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls") pod "image-registry-7cf4b5ff5-csbf4" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1") : secret "image-registry-tls" not found
May 26 11:27:52.869223 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.869188    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:52.869615 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:52.869287    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:52.869615 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.869395    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:27:52.869615 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.869408    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:27:52.869615 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.869457    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls podName:c0d91a50-338c-450a-a121-2b08b9204793 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:53.86943559 +0000 UTC m=+36.232385238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls") pod "dns-default-85gbr" (UID: "c0d91a50-338c-450a-a121-2b08b9204793") : secret "dns-default-metrics-tls" not found
May 26 11:27:52.869615 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:52.869477    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert podName:263e3474-ec60-4909-8953-24916d1eb74a nodeName:}" failed. No retries permitted until 2026-05-26 11:27:53.86946771 +0000 UTC m=+36.232417359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert") pod "ingress-canary-6d6tg" (UID: "263e3474-ec60-4909-8953-24916d1eb74a") : secret "canary-serving-cert" not found
May 26 11:27:53.536222 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:53.536167    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"]
May 26 11:27:53.539937 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:53.539897    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13850f0b_ef14_4b33_854b_943840573b6b.slice/crio-b84e65590b392da03ab94b576d7376b14f4d2e09fb22cb3536dc4294bc4dfadc WatchSource:0}: Error finding container b84e65590b392da03ab94b576d7376b14f4d2e09fb22cb3536dc4294bc4dfadc: Status 404 returned error can't find the container with id b84e65590b392da03ab94b576d7376b14f4d2e09fb22cb3536dc4294bc4dfadc
May 26 11:27:53.541546 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:53.541522    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f"]
May 26 11:27:53.545319 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:53.545277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5751358a_1f75_42f5_87e2_e09723f578e7.slice/crio-a367b85b223e0792205843b7b2566f8bb0d9aab7212aa651571a4c5d1a595012 WatchSource:0}: Error finding container a367b85b223e0792205843b7b2566f8bb0d9aab7212aa651571a4c5d1a595012: Status 404 returned error can't find the container with id a367b85b223e0792205843b7b2566f8bb0d9aab7212aa651571a4c5d1a595012
May 26 11:27:53.563009 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:53.562990    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"]
May 26 11:27:53.566631 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:27:53.566610    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfea7bbda_6d68_4d34_8b3a_3ae23d0b1b11.slice/crio-6b13ca20f2e94f184155755d51e9e9bd8d8fdf8bf4e3f3bb8d89e622743d0ea1 WatchSource:0}: Error finding container 6b13ca20f2e94f184155755d51e9e9bd8d8fdf8bf4e3f3bb8d89e622743d0ea1: Status 404 returned error can't find the container with id 6b13ca20f2e94f184155755d51e9e9bd8d8fdf8bf4e3f3bb8d89e622743d0ea1
May 26 11:27:53.777940 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:53.777905    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:53.778108 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:53.778081    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:27:53.778108 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:53.778105    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4b5ff5-csbf4: secret "image-registry-tls" not found
May 26 11:27:53.778195 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:53.778183    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls podName:455cc5be-f98e-4661-8935-bc30143a23e1 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:55.778159617 +0000 UTC m=+38.141109346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls") pod "image-registry-7cf4b5ff5-csbf4" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1") : secret "image-registry-tls" not found
May 26 11:27:53.878622 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:53.878399    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:53.878933 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:53.878645    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:53.878933 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:53.878536    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:27:53.878933 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:53.878713    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert podName:263e3474-ec60-4909-8953-24916d1eb74a nodeName:}" failed. No retries permitted until 2026-05-26 11:27:55.878695086 +0000 UTC m=+38.241644746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert") pod "ingress-canary-6d6tg" (UID: "263e3474-ec60-4909-8953-24916d1eb74a") : secret "canary-serving-cert" not found
May 26 11:27:53.878933 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:53.878730    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:27:53.878933 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:53.878766    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls podName:c0d91a50-338c-450a-a121-2b08b9204793 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:55.878756208 +0000 UTC m=+38.241705855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls") pod "dns-default-85gbr" (UID: "c0d91a50-338c-450a-a121-2b08b9204793") : secret "dns-default-metrics-tls" not found
May 26 11:27:54.371902 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:54.371827    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" event={"ID":"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11","Type":"ContainerStarted","Data":"6b13ca20f2e94f184155755d51e9e9bd8d8fdf8bf4e3f3bb8d89e622743d0ea1"}
May 26 11:27:54.372847 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:54.372825    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f" event={"ID":"5751358a-1f75-42f5-87e2-e09723f578e7","Type":"ContainerStarted","Data":"a367b85b223e0792205843b7b2566f8bb0d9aab7212aa651571a4c5d1a595012"}
May 26 11:27:54.374822 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:54.374785    2563 generic.go:358] "Generic (PLEG): container finished" podID="2a46dace-93be-4609-85f5-1682943deff4" containerID="71500d7c86c814b4cd343f3daaa206ca7edd949ac59d00d2f08e86ea6b7f470d" exitCode=0
May 26 11:27:54.374925 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:54.374854    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerDied","Data":"71500d7c86c814b4cd343f3daaa206ca7edd949ac59d00d2f08e86ea6b7f470d"}
May 26 11:27:54.375992 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:54.375697    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr" event={"ID":"13850f0b-ef14-4b33-854b-943840573b6b","Type":"ContainerStarted","Data":"b84e65590b392da03ab94b576d7376b14f4d2e09fb22cb3536dc4294bc4dfadc"}
May 26 11:27:55.091182 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.091141    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:55.105418 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.105341    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8f27af5-0b34-4fe2-ad62-afca8eb49a7c-original-pull-secret\") pod \"global-pull-secret-syncer-99q8l\" (UID: \"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c\") " pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:55.254227 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.253828    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-99q8l"
May 26 11:27:55.388627 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.387020    2563 generic.go:358] "Generic (PLEG): container finished" podID="2a46dace-93be-4609-85f5-1682943deff4" containerID="695079fc654ebb1123d9528ad528f997c8ce9cc7a58e1137994d2c2baa46f9f6" exitCode=0
May 26 11:27:55.388627 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.387100    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerDied","Data":"695079fc654ebb1123d9528ad528f997c8ce9cc7a58e1137994d2c2baa46f9f6"}
May 26 11:27:55.452596 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.452557    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-99q8l"]
May 26 11:27:55.806915 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.806230    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:55.806915 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:55.806462    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:27:55.806915 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:55.806477    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4b5ff5-csbf4: secret "image-registry-tls" not found
May 26 11:27:55.806915 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:55.806536    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls podName:455cc5be-f98e-4661-8935-bc30143a23e1 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:59.806517056 +0000 UTC m=+42.169466881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls") pod "image-registry-7cf4b5ff5-csbf4" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1") : secret "image-registry-tls" not found
May 26 11:27:55.907851 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.907568    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:55.908265 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:55.908219    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:55.908393 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:55.907758    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:27:55.908444 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:55.908426    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert podName:263e3474-ec60-4909-8953-24916d1eb74a nodeName:}" failed. No retries permitted until 2026-05-26 11:27:59.908406726 +0000 UTC m=+42.271356381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert") pod "ingress-canary-6d6tg" (UID: "263e3474-ec60-4909-8953-24916d1eb74a") : secret "canary-serving-cert" not found
May 26 11:27:55.908849 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:55.908363    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:27:55.908849 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:55.908805    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls podName:c0d91a50-338c-450a-a121-2b08b9204793 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:59.90877269 +0000 UTC m=+42.271722339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls") pod "dns-default-85gbr" (UID: "c0d91a50-338c-450a-a121-2b08b9204793") : secret "dns-default-metrics-tls" not found
May 26 11:27:56.390498 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:56.390464    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-99q8l" event={"ID":"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c","Type":"ContainerStarted","Data":"b806e196b9f1a30282129b6522a931bcba1a3e0b58895bba6b3c48c61c9fa7d5"}
May 26 11:27:56.393916 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:56.393890    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" event={"ID":"2a46dace-93be-4609-85f5-1682943deff4","Type":"ContainerStarted","Data":"80f9884d6643999775d423e69643625bc177019712dd911e62dfe9dd9b6b8248"}
May 26 11:27:56.415215 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:56.415162    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k9bkq" podStartSLOduration=5.695563041 podStartE2EDuration="38.415145873s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:27:20.847365631 +0000 UTC m=+3.210315281" lastFinishedPulling="2026-05-26 11:27:53.566948466 +0000 UTC m=+35.929898113" observedRunningTime="2026-05-26 11:27:56.413441012 +0000 UTC m=+38.776390679" watchObservedRunningTime="2026-05-26 11:27:56.415145873 +0000 UTC m=+38.778095542"
May 26 11:27:59.842228 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:59.842188    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:27:59.842698 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:59.842373    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:27:59.842698 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:59.842398    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4b5ff5-csbf4: secret "image-registry-tls" not found
May 26 11:27:59.842698 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:59.842488    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls podName:455cc5be-f98e-4661-8935-bc30143a23e1 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:07.842466235 +0000 UTC m=+50.205415882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls") pod "image-registry-7cf4b5ff5-csbf4" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1") : secret "image-registry-tls" not found
May 26 11:27:59.943171 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:59.943131    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:27:59.943348 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:27:59.943237    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:27:59.943348 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:59.943319    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:27:59.943430 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:59.943381    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:27:59.943430 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:59.943391    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls podName:c0d91a50-338c-450a-a121-2b08b9204793 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:07.9433763 +0000 UTC m=+50.306325946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls") pod "dns-default-85gbr" (UID: "c0d91a50-338c-450a-a121-2b08b9204793") : secret "dns-default-metrics-tls" not found
May 26 11:27:59.943512 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:27:59.943450    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert podName:263e3474-ec60-4909-8953-24916d1eb74a nodeName:}" failed. No retries permitted until 2026-05-26 11:28:07.943434694 +0000 UTC m=+50.306384340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert") pod "ingress-canary-6d6tg" (UID: "263e3474-ec60-4909-8953-24916d1eb74a") : secret "canary-serving-cert" not found
May 26 11:28:03.409669 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:03.409580    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f" event={"ID":"5751358a-1f75-42f5-87e2-e09723f578e7","Type":"ContainerStarted","Data":"c42a938926dfd9f048a109e3200cebe870f0aa5e11a76413478bce46b7df7f9e"}
May 26 11:28:03.410763 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:03.410738    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr" event={"ID":"13850f0b-ef14-4b33-854b-943840573b6b","Type":"ContainerStarted","Data":"e3eecd6d233b0b354bfa9ff8d49555a65806851f8d51e4974792d23c4cdc8c47"}
May 26 11:28:03.410954 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:03.410933    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:28:03.412298 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:03.412276    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" event={"ID":"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11","Type":"ContainerStarted","Data":"ae77df0f2e4d83dee9c12367eda797184d146e7b2aa8d506d9a8c99129c77869"}
May 26 11:28:03.412729 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:03.412709    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:28:03.424466 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:03.424427    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f" podStartSLOduration=28.835821095 podStartE2EDuration="38.424413241s" podCreationTimestamp="2026-05-26 11:27:25 +0000 UTC" firstStartedPulling="2026-05-26 11:27:53.546976347 +0000 UTC m=+35.909925993" lastFinishedPulling="2026-05-26 11:28:03.13556849 +0000 UTC m=+45.498518139" observedRunningTime="2026-05-26 11:28:03.423816444 +0000 UTC m=+45.786766114" watchObservedRunningTime="2026-05-26 11:28:03.424413241 +0000 UTC m=+45.787362909"
May 26 11:28:03.437849 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:03.437811    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr" podStartSLOduration=28.844447493 podStartE2EDuration="38.437783599s" podCreationTimestamp="2026-05-26 11:27:25 +0000 UTC" firstStartedPulling="2026-05-26 11:27:53.542173533 +0000 UTC m=+35.905123179" lastFinishedPulling="2026-05-26 11:28:03.135509625 +0000 UTC m=+45.498459285" observedRunningTime="2026-05-26 11:28:03.43762588 +0000 UTC m=+45.800575548" watchObservedRunningTime="2026-05-26 11:28:03.437783599 +0000 UTC m=+45.800733266"
May 26 11:28:04.415168 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:04.415122    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-99q8l" event={"ID":"b8f27af5-0b34-4fe2-ad62-afca8eb49a7c","Type":"ContainerStarted","Data":"47099e6271840a4c01a1d03b92bba6fc45a87dd5bd5be3e72e06e4ad4dc92279"}
May 26 11:28:04.430398 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:04.430358    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-99q8l" podStartSLOduration=33.411810937 podStartE2EDuration="41.430348458s" podCreationTimestamp="2026-05-26 11:27:23 +0000 UTC" firstStartedPulling="2026-05-26 11:27:55.463760877 +0000 UTC m=+37.826710527" lastFinishedPulling="2026-05-26 11:28:03.482298398 +0000 UTC m=+45.845248048" observedRunningTime="2026-05-26 11:28:04.430309673 +0000 UTC m=+46.793259340" watchObservedRunningTime="2026-05-26 11:28:04.430348458 +0000 UTC m=+46.793298119"
May 26 11:28:06.421427 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:06.421349    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" event={"ID":"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11","Type":"ContainerStarted","Data":"35fdeae3a47ea976a31912bb34cc67b8ff015d140ecdc209e88a1ac7d184d5a3"}
May 26 11:28:06.421427 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:06.421387    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" event={"ID":"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11","Type":"ContainerStarted","Data":"3e1bf71e9430e5d03214a28b1d121b1604851e33caf3df4d7fa34fc2fc23a174"}
May 26 11:28:06.439999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:06.439937    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" podStartSLOduration=28.857861079 podStartE2EDuration="41.439923741s" podCreationTimestamp="2026-05-26 11:27:25 +0000 UTC" firstStartedPulling="2026-05-26 11:27:53.568158857 +0000 UTC m=+35.931108518" lastFinishedPulling="2026-05-26 11:28:06.150221534 +0000 UTC m=+48.513171180" observedRunningTime="2026-05-26 11:28:06.439224215 +0000 UTC m=+48.802173883" watchObservedRunningTime="2026-05-26 11:28:06.439923741 +0000 UTC m=+48.802873402"
May 26 11:28:07.904375 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:07.904340    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:28:07.904832 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:07.904449    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:28:07.904832 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:07.904460    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4b5ff5-csbf4: secret "image-registry-tls" not found
May 26 11:28:07.904832 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:07.904516    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls podName:455cc5be-f98e-4661-8935-bc30143a23e1 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:23.904501953 +0000 UTC m=+66.267451610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls") pod "image-registry-7cf4b5ff5-csbf4" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1") : secret "image-registry-tls" not found
May 26 11:28:08.005560 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:08.005527    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:28:08.006068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:08.006040    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:28:08.006162 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:08.006076    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:28:08.006162 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:08.006151    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls podName:c0d91a50-338c-450a-a121-2b08b9204793 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:24.006130502 +0000 UTC m=+66.369080169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls") pod "dns-default-85gbr" (UID: "c0d91a50-338c-450a-a121-2b08b9204793") : secret "dns-default-metrics-tls" not found
May 26 11:28:08.006270 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:08.006210    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:28:08.006270 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:08.006269    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert podName:263e3474-ec60-4909-8953-24916d1eb74a nodeName:}" failed. No retries permitted until 2026-05-26 11:28:24.006245427 +0000 UTC m=+66.369195076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert") pod "ingress-canary-6d6tg" (UID: "263e3474-ec60-4909-8953-24916d1eb74a") : secret "canary-serving-cert" not found
May 26 11:28:18.371608 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:18.371583    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w89sb"
May 26 11:28:23.919711 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:23.919668    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:28:23.919711 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:23.919711    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:28:23.920181 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:23.919861    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:28:23.920181 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:23.919876    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4b5ff5-csbf4: secret "image-registry-tls" not found
May 26 11:28:23.920181 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:23.919975    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls podName:455cc5be-f98e-4661-8935-bc30143a23e1 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:55.919955082 +0000 UTC m=+98.282904733 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls") pod "image-registry-7cf4b5ff5-csbf4" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1") : secret "image-registry-tls" not found
May 26 11:28:23.922379 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:23.922362    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:28:23.931854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:23.931831    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:28:23.944238 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:23.944211    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtv2\" (UniqueName: \"kubernetes.io/projected/f4d81d1d-211d-4896-b893-c8e472dbece8-kube-api-access-jvtv2\") pod \"network-check-target-5tc55\" (UID: \"f4d81d1d-211d-4896-b893-c8e472dbece8\") " pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:28:24.020883 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:24.020857    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:28:24.020999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:24.020899    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:28:24.020999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:24.020927    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:28:24.021119 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:24.020999    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:28:24.021119 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:24.021002    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:28:24.021119 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:24.021051    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert podName:263e3474-ec60-4909-8953-24916d1eb74a nodeName:}" failed. No retries permitted until 2026-05-26 11:28:56.021027279 +0000 UTC m=+98.383976926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert") pod "ingress-canary-6d6tg" (UID: "263e3474-ec60-4909-8953-24916d1eb74a") : secret "canary-serving-cert" not found
May 26 11:28:24.021119 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:24.021069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls podName:c0d91a50-338c-450a-a121-2b08b9204793 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:56.021056498 +0000 UTC m=+98.384006144 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls") pod "dns-default-85gbr" (UID: "c0d91a50-338c-450a-a121-2b08b9204793") : secret "dns-default-metrics-tls" not found
May 26 11:28:24.023276 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:24.023261    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:28:24.031906 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:24.031883    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 11:28:24.032007 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:24.031939    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:29:28.031927308 +0000 UTC m=+130.394876954 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : secret "metrics-daemon-secret" not found
May 26 11:28:24.060459 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:24.060439    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wnncn\""
May 26 11:28:24.068877 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:24.068864    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:28:24.185558 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:24.185485    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5tc55"]
May 26 11:28:24.188989 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:28:24.188958    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d81d1d_211d_4896_b893_c8e472dbece8.slice/crio-39109d62282c242c09099e713401bbc2d89ded09a20fcdacc3666aa42d83a350 WatchSource:0}: Error finding container 39109d62282c242c09099e713401bbc2d89ded09a20fcdacc3666aa42d83a350: Status 404 returned error can't find the container with id 39109d62282c242c09099e713401bbc2d89ded09a20fcdacc3666aa42d83a350
May 26 11:28:24.467537 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:24.467461    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5tc55" event={"ID":"f4d81d1d-211d-4896-b893-c8e472dbece8","Type":"ContainerStarted","Data":"39109d62282c242c09099e713401bbc2d89ded09a20fcdacc3666aa42d83a350"}
May 26 11:28:28.478216 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:28.478173    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5tc55" event={"ID":"f4d81d1d-211d-4896-b893-c8e472dbece8","Type":"ContainerStarted","Data":"83248a1c3d1a294e76d02cc9b4403b27cced46f6a82735c32842b36cc45afc50"}
May 26 11:28:28.478572 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:28.478336    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:28:28.492634 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:28.492589    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5tc55" podStartSLOduration=67.285396822 podStartE2EDuration="1m10.492576047s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:28:24.19093223 +0000 UTC m=+66.553881877" lastFinishedPulling="2026-05-26 11:28:27.398111456 +0000 UTC m=+69.761061102" observedRunningTime="2026-05-26 11:28:28.491344222 +0000 UTC m=+70.854293890" watchObservedRunningTime="2026-05-26 11:28:28.492576047 +0000 UTC m=+70.855525716"
May 26 11:28:55.955568 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:55.955512    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:28:55.956091 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:55.955665    2563 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:28:55.956091 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:55.955683    2563 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4b5ff5-csbf4: secret "image-registry-tls" not found
May 26 11:28:55.956091 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:55.955748    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls podName:455cc5be-f98e-4661-8935-bc30143a23e1 nodeName:}" failed. No retries permitted until 2026-05-26 11:29:59.955733238 +0000 UTC m=+162.318682884 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls") pod "image-registry-7cf4b5ff5-csbf4" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1") : secret "image-registry-tls" not found
May 26 11:28:56.055929 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:56.055877    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:28:56.056091 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:56.055957    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:28:56.056091 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:56.056051    2563 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:28:56.056091 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:56.056087    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:28:56.056186 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:56.056133    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert podName:263e3474-ec60-4909-8953-24916d1eb74a nodeName:}" failed. No retries permitted until 2026-05-26 11:30:00.056115537 +0000 UTC m=+162.419065184 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert") pod "ingress-canary-6d6tg" (UID: "263e3474-ec60-4909-8953-24916d1eb74a") : secret "canary-serving-cert" not found
May 26 11:28:56.056186 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:28:56.056149    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls podName:c0d91a50-338c-450a-a121-2b08b9204793 nodeName:}" failed. No retries permitted until 2026-05-26 11:30:00.056142959 +0000 UTC m=+162.419092606 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls") pod "dns-default-85gbr" (UID: "c0d91a50-338c-450a-a121-2b08b9204793") : secret "dns-default-metrics-tls" not found
May 26 11:28:59.483099 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:28:59.483069    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5tc55"
May 26 11:29:22.960154 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:22.960124    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6t2bn_8eb48a6b-da7a-4dc2-963e-532b398e5974/dns-node-resolver/0.log"
May 26 11:29:23.760097 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:23.760052    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4bwpv_132f1313-f019-435a-98ca-62608bdbbecd/node-ca/0.log"
May 26 11:29:28.090019 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:28.089983    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:29:28.090485 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:29:28.090147    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 11:29:28.090485 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:29:28.090233    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs podName:7712945a-7f82-4e15-8b15-38ac03441d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:31:30.090215645 +0000 UTC m=+252.453165293 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs") pod "network-metrics-daemon-sp6l5" (UID: "7712945a-7f82-4e15-8b15-38ac03441d91") : secret "metrics-daemon-secret" not found
May 26 11:29:55.039199 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:29:55.039157    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4" podUID="455cc5be-f98e-4661-8935-bc30143a23e1"
May 26 11:29:55.059431 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:29:55.059407    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-6d6tg" podUID="263e3474-ec60-4909-8953-24916d1eb74a"
May 26 11:29:55.095903 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:29:55.095863    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-85gbr" podUID="c0d91a50-338c-450a-a121-2b08b9204793"
May 26 11:29:55.265211 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:29:55.265155    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sp6l5" podUID="7712945a-7f82-4e15-8b15-38ac03441d91"
May 26 11:29:55.683869 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:55.683839    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-85gbr"
May 26 11:29:55.684040 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:55.683839    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:29:56.044017 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.043987    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sh992"]
May 26 11:29:56.047189 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.047168    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.049400 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.049379    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 11:29:56.050011 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.049993    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 11:29:56.050104 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.050087    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 11:29:56.050162 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.050104    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 11:29:56.050162 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.050150    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r76dd\""
May 26 11:29:56.061415 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.061394    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sh992"]
May 26 11:29:56.101361 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.101334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92175d4e-f24c-4ac0-ab9c-ab1290c39388-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.101361 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.101361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7mt\" (UniqueName: \"kubernetes.io/projected/92175d4e-f24c-4ac0-ab9c-ab1290c39388-kube-api-access-9m7mt\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.101505 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.101384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92175d4e-f24c-4ac0-ab9c-ab1290c39388-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.101543 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.101515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92175d4e-f24c-4ac0-ab9c-ab1290c39388-data-volume\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.101575 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.101546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92175d4e-f24c-4ac0-ab9c-ab1290c39388-crio-socket\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.202166 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.202131    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92175d4e-f24c-4ac0-ab9c-ab1290c39388-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.202166 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.202166    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7mt\" (UniqueName: \"kubernetes.io/projected/92175d4e-f24c-4ac0-ab9c-ab1290c39388-kube-api-access-9m7mt\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.202366 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.202194    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92175d4e-f24c-4ac0-ab9c-ab1290c39388-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.202366 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.202285    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92175d4e-f24c-4ac0-ab9c-ab1290c39388-data-volume\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.202366 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.202319    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92175d4e-f24c-4ac0-ab9c-ab1290c39388-crio-socket\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.202504 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.202403    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92175d4e-f24c-4ac0-ab9c-ab1290c39388-crio-socket\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.202675 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.202652    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92175d4e-f24c-4ac0-ab9c-ab1290c39388-data-volume\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.202781 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.202673    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92175d4e-f24c-4ac0-ab9c-ab1290c39388-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.204764 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.204743    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92175d4e-f24c-4ac0-ab9c-ab1290c39388-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.220868 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.220845    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7mt\" (UniqueName: \"kubernetes.io/projected/92175d4e-f24c-4ac0-ab9c-ab1290c39388-kube-api-access-9m7mt\") pod \"insights-runtime-extractor-sh992\" (UID: \"92175d4e-f24c-4ac0-ab9c-ab1290c39388\") " pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.355679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.355600    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sh992"
May 26 11:29:56.473919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.473889    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sh992"]
May 26 11:29:56.479176 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:29:56.479150    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92175d4e_f24c_4ac0_ab9c_ab1290c39388.slice/crio-3e1cc9cf5680475a90448abc6766a18550cc670c65d02d4e269dfab83b7e9123 WatchSource:0}: Error finding container 3e1cc9cf5680475a90448abc6766a18550cc670c65d02d4e269dfab83b7e9123: Status 404 returned error can't find the container with id 3e1cc9cf5680475a90448abc6766a18550cc670c65d02d4e269dfab83b7e9123
May 26 11:29:56.687437 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.687355    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh992" event={"ID":"92175d4e-f24c-4ac0-ab9c-ab1290c39388","Type":"ContainerStarted","Data":"233cf6f0e596d39b359161efad4960b09119af6d066262931228695f9a1de3a9"}
May 26 11:29:56.687437 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:56.687391    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh992" event={"ID":"92175d4e-f24c-4ac0-ab9c-ab1290c39388","Type":"ContainerStarted","Data":"3e1cc9cf5680475a90448abc6766a18550cc670c65d02d4e269dfab83b7e9123"}
May 26 11:29:57.691089 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:57.691049    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh992" event={"ID":"92175d4e-f24c-4ac0-ab9c-ab1290c39388","Type":"ContainerStarted","Data":"c53188665710fbe72d50986a05fa65dd8e34e704ef19e60e2592294592f997b7"}
May 26 11:29:59.696974 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:59.696931    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh992" event={"ID":"92175d4e-f24c-4ac0-ab9c-ab1290c39388","Type":"ContainerStarted","Data":"84d08afdb6c37554eb16983cdfcef4f14b1856e4702884c2c064a282386e2544"}
May 26 11:29:59.714157 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:29:59.714107    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sh992" podStartSLOduration=1.596284918 podStartE2EDuration="3.714093977s" podCreationTimestamp="2026-05-26 11:29:56 +0000 UTC" firstStartedPulling="2026-05-26 11:29:56.534206551 +0000 UTC m=+158.897156202" lastFinishedPulling="2026-05-26 11:29:58.652015612 +0000 UTC m=+161.014965261" observedRunningTime="2026-05-26 11:29:59.713020716 +0000 UTC m=+162.075970385" watchObservedRunningTime="2026-05-26 11:29:59.714093977 +0000 UTC m=+162.077043645"
May 26 11:30:00.032518 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.032434    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:30:00.034918 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.034896    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"image-registry-7cf4b5ff5-csbf4\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") " pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:30:00.132920 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.132883    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:30:00.132920 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.132927    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:30:00.135325 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.135302    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d91a50-338c-450a-a121-2b08b9204793-metrics-tls\") pod \"dns-default-85gbr\" (UID: \"c0d91a50-338c-450a-a121-2b08b9204793\") " pod="openshift-dns/dns-default-85gbr"
May 26 11:30:00.135435 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.135408    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/263e3474-ec60-4909-8953-24916d1eb74a-cert\") pod \"ingress-canary-6d6tg\" (UID: \"263e3474-ec60-4909-8953-24916d1eb74a\") " pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:30:00.186622 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.186596    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sqkc6\""
May 26 11:30:00.186622 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.186596    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-697bc\""
May 26 11:30:00.194864 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.194845    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:30:00.194947 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.194932    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-85gbr"
May 26 11:30:00.331394 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.331362    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf4b5ff5-csbf4"]
May 26 11:30:00.335178 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:30:00.335153    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455cc5be_f98e_4661_8935_bc30143a23e1.slice/crio-5299252a95e2e0f8e030b66c9a521e9c507710f5fabe1111716139a6d742837b WatchSource:0}: Error finding container 5299252a95e2e0f8e030b66c9a521e9c507710f5fabe1111716139a6d742837b: Status 404 returned error can't find the container with id 5299252a95e2e0f8e030b66c9a521e9c507710f5fabe1111716139a6d742837b
May 26 11:30:00.343094 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.343073    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-85gbr"]
May 26 11:30:00.347176 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:30:00.347155    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d91a50_338c_450a_a121_2b08b9204793.slice/crio-1aaa9d4fd9e5a9cfa92f114e312c237a4cae308e5ed111f11870e36d927124a8 WatchSource:0}: Error finding container 1aaa9d4fd9e5a9cfa92f114e312c237a4cae308e5ed111f11870e36d927124a8: Status 404 returned error can't find the container with id 1aaa9d4fd9e5a9cfa92f114e312c237a4cae308e5ed111f11870e36d927124a8
May 26 11:30:00.701055 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.701017    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4" event={"ID":"455cc5be-f98e-4661-8935-bc30143a23e1","Type":"ContainerStarted","Data":"1dd95cef3e219b61bea4d65fc342000ea9999e103911e2aace2d9707803468d8"}
May 26 11:30:00.701512 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.701060    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4" event={"ID":"455cc5be-f98e-4661-8935-bc30143a23e1","Type":"ContainerStarted","Data":"5299252a95e2e0f8e030b66c9a521e9c507710f5fabe1111716139a6d742837b"}
May 26 11:30:00.701512 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.701099    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:30:00.702072 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.702048    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-85gbr" event={"ID":"c0d91a50-338c-450a-a121-2b08b9204793","Type":"ContainerStarted","Data":"1aaa9d4fd9e5a9cfa92f114e312c237a4cae308e5ed111f11870e36d927124a8"}
May 26 11:30:00.720117 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:00.720073    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4" podStartSLOduration=162.720058268 podStartE2EDuration="2m42.720058268s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:30:00.719334324 +0000 UTC m=+163.082283994" watchObservedRunningTime="2026-05-26 11:30:00.720058268 +0000 UTC m=+163.083007938"
May 26 11:30:02.709005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:02.708965    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-85gbr" event={"ID":"c0d91a50-338c-450a-a121-2b08b9204793","Type":"ContainerStarted","Data":"1fe380cf700a6342623b984fcf6454c37badd9ac10779a2b16036457b6e4c914"}
May 26 11:30:02.709005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:02.709004    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-85gbr" event={"ID":"c0d91a50-338c-450a-a121-2b08b9204793","Type":"ContainerStarted","Data":"eb6d0093d3348a4619f55cf882657674b8c9e54563e17ad36e7457f8f766a486"}
May 26 11:30:02.709421 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:02.709095    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-85gbr"
May 26 11:30:02.724784 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:02.724725    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-85gbr" podStartSLOduration=129.22017567 podStartE2EDuration="2m10.724710698s" podCreationTimestamp="2026-05-26 11:27:52 +0000 UTC" firstStartedPulling="2026-05-26 11:30:00.348862638 +0000 UTC m=+162.711812289" lastFinishedPulling="2026-05-26 11:30:01.853397667 +0000 UTC m=+164.216347317" observedRunningTime="2026-05-26 11:30:02.724102995 +0000 UTC m=+165.087052664" watchObservedRunningTime="2026-05-26 11:30:02.724710698 +0000 UTC m=+165.087660368"
May 26 11:30:03.411760 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.411723    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr" podUID="13850f0b-ef14-4b33-854b-943840573b6b" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.6:8000/readyz\": dial tcp 10.132.0.6:8000: connect: connection refused"
May 26 11:30:03.713179 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.713097    2563 generic.go:358] "Generic (PLEG): container finished" podID="13850f0b-ef14-4b33-854b-943840573b6b" containerID="e3eecd6d233b0b354bfa9ff8d49555a65806851f8d51e4974792d23c4cdc8c47" exitCode=1
May 26 11:30:03.713179 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.713170    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr" event={"ID":"13850f0b-ef14-4b33-854b-943840573b6b","Type":"ContainerDied","Data":"e3eecd6d233b0b354bfa9ff8d49555a65806851f8d51e4974792d23c4cdc8c47"}
May 26 11:30:03.713609 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.713517    2563 scope.go:117] "RemoveContainer" containerID="e3eecd6d233b0b354bfa9ff8d49555a65806851f8d51e4974792d23c4cdc8c47"
May 26 11:30:03.714401 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.714384    2563 generic.go:358] "Generic (PLEG): container finished" podID="5751358a-1f75-42f5-87e2-e09723f578e7" containerID="c42a938926dfd9f048a109e3200cebe870f0aa5e11a76413478bce46b7df7f9e" exitCode=255
May 26 11:30:03.714467 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.714451    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f" event={"ID":"5751358a-1f75-42f5-87e2-e09723f578e7","Type":"ContainerDied","Data":"c42a938926dfd9f048a109e3200cebe870f0aa5e11a76413478bce46b7df7f9e"}
May 26 11:30:03.714898 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.714884    2563 scope.go:117] "RemoveContainer" containerID="c42a938926dfd9f048a109e3200cebe870f0aa5e11a76413478bce46b7df7f9e"
May 26 11:30:03.791619 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.791587    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zdkbk"]
May 26 11:30:03.795330 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.795305    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.797478 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.797196    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 11:30:03.797760 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.797734    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 11:30:03.797760 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.797755    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 11:30:03.797954 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.797736    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k2glr\""
May 26 11:30:03.797954 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.797830    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 11:30:03.798181 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.798159    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 11:30:03.798260 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.798195    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 11:30:03.860476 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89wn\" (UniqueName: \"kubernetes.io/projected/6479af9e-2fae-4bcf-bcad-bb978c70c652-kube-api-access-f89wn\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.860658 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.860658 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-sys\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.860658 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-root\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.860862 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.860862 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-wtmp\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.860862 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-tls\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.860862 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6479af9e-2fae-4bcf-bcad-bb978c70c652-metrics-client-ca\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.860862 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.860861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-textfile\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.961862 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.961823    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-sys\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962031 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.961869    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-root\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962031 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.961889    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962031 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.961908    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-wtmp\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962031 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.961920    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-sys\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962031 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.961937    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-tls\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962031 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.961974    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6479af9e-2fae-4bcf-bcad-bb978c70c652-metrics-client-ca\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962031 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.962006    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-textfile\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962031 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.962017    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-root\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962411 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.962047    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f89wn\" (UniqueName: \"kubernetes.io/projected/6479af9e-2fae-4bcf-bcad-bb978c70c652-kube-api-access-f89wn\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962411 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.962054    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-wtmp\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962411 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:30:03.962111    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 11:30:03.962411 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.962154    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962411 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:30:03.962166    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-tls podName:6479af9e-2fae-4bcf-bcad-bb978c70c652 nodeName:}" failed. No retries permitted until 2026-05-26 11:30:04.462146577 +0000 UTC m=+166.825096239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-tls") pod "node-exporter-zdkbk" (UID: "6479af9e-2fae-4bcf-bcad-bb978c70c652") : secret "node-exporter-tls" not found
May 26 11:30:03.962411 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.962376    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-textfile\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962704 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.962617    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.962704 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.962663    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6479af9e-2fae-4bcf-bcad-bb978c70c652-metrics-client-ca\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.964341 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.964301    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:03.970255 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:03.970230    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89wn\" (UniqueName: \"kubernetes.io/projected/6479af9e-2fae-4bcf-bcad-bb978c70c652-kube-api-access-f89wn\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:04.467324 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:04.467289    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-tls\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:04.469751 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:04.469727    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6479af9e-2fae-4bcf-bcad-bb978c70c652-node-exporter-tls\") pod \"node-exporter-zdkbk\" (UID: \"6479af9e-2fae-4bcf-bcad-bb978c70c652\") " pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:04.706174 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:04.706123    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zdkbk"
May 26 11:30:04.714698 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:30:04.714664    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6479af9e_2fae_4bcf_bcad_bb978c70c652.slice/crio-78e795838a81bb232d9f38fb2a3c7c0037c805b4d1c17392912d34ea79c35388 WatchSource:0}: Error finding container 78e795838a81bb232d9f38fb2a3c7c0037c805b4d1c17392912d34ea79c35388: Status 404 returned error can't find the container with id 78e795838a81bb232d9f38fb2a3c7c0037c805b4d1c17392912d34ea79c35388
May 26 11:30:04.720179 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:04.719437    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774f68f574-xpv4f" event={"ID":"5751358a-1f75-42f5-87e2-e09723f578e7","Type":"ContainerStarted","Data":"2af0de6de0d9939add3582c69e7c4fd6d596e7954ad79114068ecbf587aa7801"}
May 26 11:30:04.721397 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:04.721373    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr" event={"ID":"13850f0b-ef14-4b33-854b-943840573b6b","Type":"ContainerStarted","Data":"ec85393b85b0e447c8d065d8c85c77a7ceb077d793a5e653a9feb2480d8f4b21"}
May 26 11:30:04.721652 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:04.721629    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:30:04.722293 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:04.722264    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9dc6d46-flvgr"
May 26 11:30:05.724767 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:05.724691    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdkbk" event={"ID":"6479af9e-2fae-4bcf-bcad-bb978c70c652","Type":"ContainerStarted","Data":"78e795838a81bb232d9f38fb2a3c7c0037c805b4d1c17392912d34ea79c35388"}
May 26 11:30:06.728691 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:06.728659    2563 generic.go:358] "Generic (PLEG): container finished" podID="6479af9e-2fae-4bcf-bcad-bb978c70c652" containerID="ac268527b2dec9c7164b00dcadee84da63e4597aa46e325fc9b907821ddcc5c3" exitCode=0
May 26 11:30:06.729172 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:06.728742    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdkbk" event={"ID":"6479af9e-2fae-4bcf-bcad-bb978c70c652","Type":"ContainerDied","Data":"ac268527b2dec9c7164b00dcadee84da63e4597aa46e325fc9b907821ddcc5c3"}
May 26 11:30:07.238402 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:07.238363    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:30:07.240547 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:07.240531    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gmhr4\""
May 26 11:30:07.249018 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:07.248990    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6d6tg"
May 26 11:30:07.369226 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:07.369196    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6d6tg"]
May 26 11:30:07.372100 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:30:07.372071    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263e3474_ec60_4909_8953_24916d1eb74a.slice/crio-3ebc642d9752dfd1e011be653b7d4c57bf07b057ec4f82d83091ee5da5007bae WatchSource:0}: Error finding container 3ebc642d9752dfd1e011be653b7d4c57bf07b057ec4f82d83091ee5da5007bae: Status 404 returned error can't find the container with id 3ebc642d9752dfd1e011be653b7d4c57bf07b057ec4f82d83091ee5da5007bae
May 26 11:30:07.733172 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:07.733125    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdkbk" event={"ID":"6479af9e-2fae-4bcf-bcad-bb978c70c652","Type":"ContainerStarted","Data":"7164ad441ae34eec24834b6aaee07ae32372f1ddf7e25cdc44d952403b0996f4"}
May 26 11:30:07.733172 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:07.733172    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdkbk" event={"ID":"6479af9e-2fae-4bcf-bcad-bb978c70c652","Type":"ContainerStarted","Data":"90ab3740159a3c6ea6bcc7f491c3181053a0258a0cb4db6202e13833df640d20"}
May 26 11:30:07.734095 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:07.734074    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6d6tg" event={"ID":"263e3474-ec60-4909-8953-24916d1eb74a","Type":"ContainerStarted","Data":"3ebc642d9752dfd1e011be653b7d4c57bf07b057ec4f82d83091ee5da5007bae"}
May 26 11:30:07.752358 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:07.752314    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zdkbk" podStartSLOduration=3.721408784 podStartE2EDuration="4.752300892s" podCreationTimestamp="2026-05-26 11:30:03 +0000 UTC" firstStartedPulling="2026-05-26 11:30:04.716742861 +0000 UTC m=+167.079692512" lastFinishedPulling="2026-05-26 11:30:05.747634972 +0000 UTC m=+168.110584620" observedRunningTime="2026-05-26 11:30:07.75139967 +0000 UTC m=+170.114349339" watchObservedRunningTime="2026-05-26 11:30:07.752300892 +0000 UTC m=+170.115250557"
May 26 11:30:09.740728 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:09.740689    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6d6tg" event={"ID":"263e3474-ec60-4909-8953-24916d1eb74a","Type":"ContainerStarted","Data":"df0a10091c2e7fd62f9cd7fffaadc21253068e5e8dac8a4981ffd7aaa523421d"}
May 26 11:30:09.753184 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:09.753124    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6d6tg" podStartSLOduration=137.043333597 podStartE2EDuration="2m18.753109891s" podCreationTimestamp="2026-05-26 11:27:51 +0000 UTC" firstStartedPulling="2026-05-26 11:30:07.37414683 +0000 UTC m=+169.737096476" lastFinishedPulling="2026-05-26 11:30:09.083923123 +0000 UTC m=+171.446872770" observedRunningTime="2026-05-26 11:30:09.752835369 +0000 UTC m=+172.115785040" watchObservedRunningTime="2026-05-26 11:30:09.753109891 +0000 UTC m=+172.116059565"
May 26 11:30:10.238870 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:10.238828    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:30:12.717139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:12.717110    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-85gbr"
May 26 11:30:18.643324 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:18.643281    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cf4b5ff5-csbf4"]
May 26 11:30:18.647395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:18.647365    2563 patch_prober.go:28] interesting pod/image-registry-7cf4b5ff5-csbf4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 11:30:18.647524 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:18.647448    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4" podUID="455cc5be-f98e-4661-8935-bc30143a23e1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:30:28.648364 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:28.648333    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:30:43.662303 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.662243    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4" podUID="455cc5be-f98e-4661-8935-bc30143a23e1" containerName="registry" containerID="cri-o://1dd95cef3e219b61bea4d65fc342000ea9999e103911e2aace2d9707803468d8" gracePeriod=30
May 26 11:30:43.828564 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.828527    2563 generic.go:358] "Generic (PLEG): container finished" podID="455cc5be-f98e-4661-8935-bc30143a23e1" containerID="1dd95cef3e219b61bea4d65fc342000ea9999e103911e2aace2d9707803468d8" exitCode=0
May 26 11:30:43.828564 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.828560    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4" event={"ID":"455cc5be-f98e-4661-8935-bc30143a23e1","Type":"ContainerDied","Data":"1dd95cef3e219b61bea4d65fc342000ea9999e103911e2aace2d9707803468d8"}
May 26 11:30:43.904398 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.904374    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:30:43.973841 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.973734    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-registry-certificates\") pod \"455cc5be-f98e-4661-8935-bc30143a23e1\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") "
May 26 11:30:43.973841 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.973786    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") pod \"455cc5be-f98e-4661-8935-bc30143a23e1\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") "
May 26 11:30:43.973841 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.973826    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-installation-pull-secrets\") pod \"455cc5be-f98e-4661-8935-bc30143a23e1\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") "
May 26 11:30:43.974166 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.973939    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-image-registry-private-configuration\") pod \"455cc5be-f98e-4661-8935-bc30143a23e1\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") "
May 26 11:30:43.974166 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.974003    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhp8w\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-kube-api-access-zhp8w\") pod \"455cc5be-f98e-4661-8935-bc30143a23e1\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") "
May 26 11:30:43.974166 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.974035    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-bound-sa-token\") pod \"455cc5be-f98e-4661-8935-bc30143a23e1\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") "
May 26 11:30:43.974166 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.974084    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/455cc5be-f98e-4661-8935-bc30143a23e1-ca-trust-extracted\") pod \"455cc5be-f98e-4661-8935-bc30143a23e1\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") "
May 26 11:30:43.974166 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.974113    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-trusted-ca\") pod \"455cc5be-f98e-4661-8935-bc30143a23e1\" (UID: \"455cc5be-f98e-4661-8935-bc30143a23e1\") "
May 26 11:30:43.974416 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.974158    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "455cc5be-f98e-4661-8935-bc30143a23e1" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:30:43.974703 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.974487    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-registry-certificates\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:30:43.974703 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.974566    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "455cc5be-f98e-4661-8935-bc30143a23e1" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:30:43.976668 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.976632    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-kube-api-access-zhp8w" (OuterVolumeSpecName: "kube-api-access-zhp8w") pod "455cc5be-f98e-4661-8935-bc30143a23e1" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1"). InnerVolumeSpecName "kube-api-access-zhp8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:30:43.976786 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.976686    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "455cc5be-f98e-4661-8935-bc30143a23e1" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:30:43.976786 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.976726    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "455cc5be-f98e-4661-8935-bc30143a23e1" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:30:43.976903 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.976853    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "455cc5be-f98e-4661-8935-bc30143a23e1" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:30:43.976943 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.976896    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "455cc5be-f98e-4661-8935-bc30143a23e1" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:30:43.982901 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:43.982876    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455cc5be-f98e-4661-8935-bc30143a23e1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "455cc5be-f98e-4661-8935-bc30143a23e1" (UID: "455cc5be-f98e-4661-8935-bc30143a23e1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:30:44.075504 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.075450    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-registry-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:30:44.075504 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.075494    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-installation-pull-secrets\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:30:44.075504 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.075507    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/455cc5be-f98e-4661-8935-bc30143a23e1-image-registry-private-configuration\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:30:44.075504 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.075517    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhp8w\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-kube-api-access-zhp8w\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:30:44.075504 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.075527    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/455cc5be-f98e-4661-8935-bc30143a23e1-bound-sa-token\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:30:44.075837 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.075536    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/455cc5be-f98e-4661-8935-bc30143a23e1-ca-trust-extracted\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:30:44.075837 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.075544    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/455cc5be-f98e-4661-8935-bc30143a23e1-trusted-ca\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:30:44.832632 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.832595    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4" event={"ID":"455cc5be-f98e-4661-8935-bc30143a23e1","Type":"ContainerDied","Data":"5299252a95e2e0f8e030b66c9a521e9c507710f5fabe1111716139a6d742837b"}
May 26 11:30:44.832632 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.832637    2563 scope.go:117] "RemoveContainer" containerID="1dd95cef3e219b61bea4d65fc342000ea9999e103911e2aace2d9707803468d8"
May 26 11:30:44.833139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.832659    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4b5ff5-csbf4"
May 26 11:30:44.847936 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.847911    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cf4b5ff5-csbf4"]
May 26 11:30:44.851332 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:44.851310    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cf4b5ff5-csbf4"]
May 26 11:30:46.241827 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:46.241778    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455cc5be-f98e-4661-8935-bc30143a23e1" path="/var/lib/kubelet/pods/455cc5be-f98e-4661-8935-bc30143a23e1/volumes"
May 26 11:30:52.347670 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:30:52.347633    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" podUID="fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:31:02.347532 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:02.347493    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" podUID="fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:31:12.347921 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:12.347880    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" podUID="fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:31:12.348309 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:12.347964    2563 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg"
May 26 11:31:12.348421 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:12.348385    2563 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"35fdeae3a47ea976a31912bb34cc67b8ff015d140ecdc209e88a1ac7d184d5a3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 26 11:31:12.348465 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:12.348443    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" podUID="fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11" containerName="service-proxy" containerID="cri-o://35fdeae3a47ea976a31912bb34cc67b8ff015d140ecdc209e88a1ac7d184d5a3" gracePeriod=30
May 26 11:31:12.908485 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:12.908441    2563 generic.go:358] "Generic (PLEG): container finished" podID="fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11" containerID="35fdeae3a47ea976a31912bb34cc67b8ff015d140ecdc209e88a1ac7d184d5a3" exitCode=2
May 26 11:31:12.908647 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:12.908504    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" event={"ID":"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11","Type":"ContainerDied","Data":"35fdeae3a47ea976a31912bb34cc67b8ff015d140ecdc209e88a1ac7d184d5a3"}
May 26 11:31:12.908647 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:12.908538    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-666fd97845-2zbgg" event={"ID":"fea7bbda-6d68-4d34-8b3a-3ae23d0b1b11","Type":"ContainerStarted","Data":"b0c81c5476e02bec55efc7afbcb3c730416bb3a2a63f9d8d442fa6e713825860"}
May 26 11:31:30.123127 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:30.123084    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:31:30.125521 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:30.125500    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7712945a-7f82-4e15-8b15-38ac03441d91-metrics-certs\") pod \"network-metrics-daemon-sp6l5\" (UID: \"7712945a-7f82-4e15-8b15-38ac03441d91\") " pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:31:30.341877 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:30.341847    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jz9wk\""
May 26 11:31:30.349858 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:30.349840    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sp6l5"
May 26 11:31:30.467653 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:30.467625    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sp6l5"]
May 26 11:31:30.470450 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:31:30.470421    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7712945a_7f82_4e15_8b15_38ac03441d91.slice/crio-ca40d6364c9e37f65fd442d31a3ee013ef30c00e094d54ac56fe3018f139dea9 WatchSource:0}: Error finding container ca40d6364c9e37f65fd442d31a3ee013ef30c00e094d54ac56fe3018f139dea9: Status 404 returned error can't find the container with id ca40d6364c9e37f65fd442d31a3ee013ef30c00e094d54ac56fe3018f139dea9
May 26 11:31:30.956165 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:30.956132    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sp6l5" event={"ID":"7712945a-7f82-4e15-8b15-38ac03441d91","Type":"ContainerStarted","Data":"ca40d6364c9e37f65fd442d31a3ee013ef30c00e094d54ac56fe3018f139dea9"}
May 26 11:31:31.960596 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:31.960498    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sp6l5" event={"ID":"7712945a-7f82-4e15-8b15-38ac03441d91","Type":"ContainerStarted","Data":"cae0581b20f076f899632e4e55a0f9e1bf8b99612f78cb81918f7a3f953b2c2f"}
May 26 11:31:31.960596 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:31.960532    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sp6l5" event={"ID":"7712945a-7f82-4e15-8b15-38ac03441d91","Type":"ContainerStarted","Data":"b133e44e37e2d48c20a4954ec53233159b80640ebd45bf0c8d0b4a1d54831c75"}
May 26 11:31:31.976688 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:31:31.976640    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sp6l5" podStartSLOduration=252.823132187 podStartE2EDuration="4m13.976624889s" podCreationTimestamp="2026-05-26 11:27:18 +0000 UTC" firstStartedPulling="2026-05-26 11:31:30.47237094 +0000 UTC m=+252.835320593" lastFinishedPulling="2026-05-26 11:31:31.625863646 +0000 UTC m=+253.988813295" observedRunningTime="2026-05-26 11:31:31.976319967 +0000 UTC m=+254.339269636" watchObservedRunningTime="2026-05-26 11:31:31.976624889 +0000 UTC m=+254.339574560"
May 26 11:32:18.118810 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:32:18.118769    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:32:18.119361 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:32:18.119255    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:32:18.123584 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:32:18.123564    2563 kubelet.go:1628] "Image garbage collection succeeded"
May 26 11:37:18.134232 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:37:18.134200    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:37:18.136073 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:37:18.136050    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:40:52.537043 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.537010    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-279st"]
May 26 11:40:52.537462 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.537224    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455cc5be-f98e-4661-8935-bc30143a23e1" containerName="registry"
May 26 11:40:52.537462 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.537235    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="455cc5be-f98e-4661-8935-bc30143a23e1" containerName="registry"
May 26 11:40:52.537462 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.537284    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="455cc5be-f98e-4661-8935-bc30143a23e1" containerName="registry"
May 26 11:40:52.538762 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.538745    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:52.541032 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.541010    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 11:40:52.541032 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.541023    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 11:40:52.541290 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.541273    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 11:40:52.541501 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.541485    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4zcqw\""
May 26 11:40:52.549025 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.549006    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-279st"]
May 26 11:40:52.613957 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.613920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsshf\" (UniqueName: \"kubernetes.io/projected/1d3531b3-ada3-4f07-ae52-3c63415bad4e-kube-api-access-gsshf\") pod \"seaweedfs-79f457656b-279st\" (UID: \"1d3531b3-ada3-4f07-ae52-3c63415bad4e\") " pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:52.614119 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.613971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d3531b3-ada3-4f07-ae52-3c63415bad4e-data\") pod \"seaweedfs-79f457656b-279st\" (UID: \"1d3531b3-ada3-4f07-ae52-3c63415bad4e\") " pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:52.714314 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.714273    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gsshf\" (UniqueName: \"kubernetes.io/projected/1d3531b3-ada3-4f07-ae52-3c63415bad4e-kube-api-access-gsshf\") pod \"seaweedfs-79f457656b-279st\" (UID: \"1d3531b3-ada3-4f07-ae52-3c63415bad4e\") " pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:52.714443 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.714335    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d3531b3-ada3-4f07-ae52-3c63415bad4e-data\") pod \"seaweedfs-79f457656b-279st\" (UID: \"1d3531b3-ada3-4f07-ae52-3c63415bad4e\") " pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:52.714662 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.714646    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d3531b3-ada3-4f07-ae52-3c63415bad4e-data\") pod \"seaweedfs-79f457656b-279st\" (UID: \"1d3531b3-ada3-4f07-ae52-3c63415bad4e\") " pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:52.723516 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.723487    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsshf\" (UniqueName: \"kubernetes.io/projected/1d3531b3-ada3-4f07-ae52-3c63415bad4e-kube-api-access-gsshf\") pod \"seaweedfs-79f457656b-279st\" (UID: \"1d3531b3-ada3-4f07-ae52-3c63415bad4e\") " pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:52.847711 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.847679    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:52.972219 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.972186    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-279st"]
May 26 11:40:52.975174 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:40:52.975149    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3531b3_ada3_4f07_ae52_3c63415bad4e.slice/crio-31123801d41153c42ea5f695a69522596e9fab180b71569cdb9efc933bc77686 WatchSource:0}: Error finding container 31123801d41153c42ea5f695a69522596e9fab180b71569cdb9efc933bc77686: Status 404 returned error can't find the container with id 31123801d41153c42ea5f695a69522596e9fab180b71569cdb9efc933bc77686
May 26 11:40:52.976419 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:52.976402    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:40:53.339546 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:53.339511    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-279st" event={"ID":"1d3531b3-ada3-4f07-ae52-3c63415bad4e","Type":"ContainerStarted","Data":"31123801d41153c42ea5f695a69522596e9fab180b71569cdb9efc933bc77686"}
May 26 11:40:56.349692 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:56.349617    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-279st" event={"ID":"1d3531b3-ada3-4f07-ae52-3c63415bad4e","Type":"ContainerStarted","Data":"6eaa953383d64d80f9eb57dd092faf735454b2848eb90abcf5e7c54d07d19225"}
May 26 11:40:56.350069 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:56.349728    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:40:56.366411 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:40:56.366365    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-279st" podStartSLOduration=1.296785873 podStartE2EDuration="4.3663522s" podCreationTimestamp="2026-05-26 11:40:52 +0000 UTC" firstStartedPulling="2026-05-26 11:40:52.976520673 +0000 UTC m=+815.339470321" lastFinishedPulling="2026-05-26 11:40:56.046086982 +0000 UTC m=+818.409036648" observedRunningTime="2026-05-26 11:40:56.36550491 +0000 UTC m=+818.728454579" watchObservedRunningTime="2026-05-26 11:40:56.3663522 +0000 UTC m=+818.729301868"
May 26 11:41:02.353916 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:02.353889    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-279st"
May 26 11:41:28.826410 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.826376    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-wnslt"]
May 26 11:41:28.829004 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.828980    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:28.831660 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.831640    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-6qj6b\""
May 26 11:41:28.831757 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.831673    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 26 11:41:28.839966 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.839948    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-wnslt"]
May 26 11:41:28.866774 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.866736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975qv\" (UniqueName: \"kubernetes.io/projected/e9f537db-2d0b-4a94-a27d-ec5c0baa569a-kube-api-access-975qv\") pod \"kserve-controller-manager-7cd9b78f5d-wnslt\" (UID: \"e9f537db-2d0b-4a94-a27d-ec5c0baa569a\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:28.866931 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.866858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f537db-2d0b-4a94-a27d-ec5c0baa569a-cert\") pod \"kserve-controller-manager-7cd9b78f5d-wnslt\" (UID: \"e9f537db-2d0b-4a94-a27d-ec5c0baa569a\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:28.968055 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.968023    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-975qv\" (UniqueName: \"kubernetes.io/projected/e9f537db-2d0b-4a94-a27d-ec5c0baa569a-kube-api-access-975qv\") pod \"kserve-controller-manager-7cd9b78f5d-wnslt\" (UID: \"e9f537db-2d0b-4a94-a27d-ec5c0baa569a\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:28.968200 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.968105    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f537db-2d0b-4a94-a27d-ec5c0baa569a-cert\") pod \"kserve-controller-manager-7cd9b78f5d-wnslt\" (UID: \"e9f537db-2d0b-4a94-a27d-ec5c0baa569a\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:28.970642 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.970623    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f537db-2d0b-4a94-a27d-ec5c0baa569a-cert\") pod \"kserve-controller-manager-7cd9b78f5d-wnslt\" (UID: \"e9f537db-2d0b-4a94-a27d-ec5c0baa569a\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:28.976089 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:28.976067    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-975qv\" (UniqueName: \"kubernetes.io/projected/e9f537db-2d0b-4a94-a27d-ec5c0baa569a-kube-api-access-975qv\") pod \"kserve-controller-manager-7cd9b78f5d-wnslt\" (UID: \"e9f537db-2d0b-4a94-a27d-ec5c0baa569a\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:29.138132 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:29.138050    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:29.262835 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:29.262783    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-wnslt"]
May 26 11:41:29.265682 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:41:29.265652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f537db_2d0b_4a94_a27d_ec5c0baa569a.slice/crio-b6c962e8d2ae25107bda2339a12611c61955e97cbff93693494f8f456a07b235 WatchSource:0}: Error finding container b6c962e8d2ae25107bda2339a12611c61955e97cbff93693494f8f456a07b235: Status 404 returned error can't find the container with id b6c962e8d2ae25107bda2339a12611c61955e97cbff93693494f8f456a07b235
May 26 11:41:29.435951 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:29.435869    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt" event={"ID":"e9f537db-2d0b-4a94-a27d-ec5c0baa569a","Type":"ContainerStarted","Data":"b6c962e8d2ae25107bda2339a12611c61955e97cbff93693494f8f456a07b235"}
May 26 11:41:32.444922 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:32.444885    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt" event={"ID":"e9f537db-2d0b-4a94-a27d-ec5c0baa569a","Type":"ContainerStarted","Data":"2f2e8ecc1a9239defece713f0b2326e4b88b358dbba20131f042d8c1005dc425"}
May 26 11:41:32.445292 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:32.444942    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:41:32.462381 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:41:32.462329    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt" podStartSLOduration=2.026082339 podStartE2EDuration="4.462313047s" podCreationTimestamp="2026-05-26 11:41:28 +0000 UTC" firstStartedPulling="2026-05-26 11:41:29.267004234 +0000 UTC m=+851.629953881" lastFinishedPulling="2026-05-26 11:41:31.703234935 +0000 UTC m=+854.066184589" observedRunningTime="2026-05-26 11:41:32.461335638 +0000 UTC m=+854.824285307" watchObservedRunningTime="2026-05-26 11:41:32.462313047 +0000 UTC m=+854.825262718"
May 26 11:42:03.451783 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:03.451748    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-wnslt"
May 26 11:42:04.352104 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.352073    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-2khfb"]
May 26 11:42:04.355173 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.355152    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:04.357919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.357900    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 26 11:42:04.358459 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.358443    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-95zsh\""
May 26 11:42:04.366144 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.366119    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-2khfb"]
May 26 11:42:04.526862 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.526832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd8a2ed-f55f-409c-956b-90085f39a31c-tls-certs\") pod \"model-serving-api-86f7b4b499-2khfb\" (UID: \"7dd8a2ed-f55f-409c-956b-90085f39a31c\") " pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:04.527343 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.526870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85kg\" (UniqueName: \"kubernetes.io/projected/7dd8a2ed-f55f-409c-956b-90085f39a31c-kube-api-access-n85kg\") pod \"model-serving-api-86f7b4b499-2khfb\" (UID: \"7dd8a2ed-f55f-409c-956b-90085f39a31c\") " pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:04.627354 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.627268    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd8a2ed-f55f-409c-956b-90085f39a31c-tls-certs\") pod \"model-serving-api-86f7b4b499-2khfb\" (UID: \"7dd8a2ed-f55f-409c-956b-90085f39a31c\") " pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:04.627354 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.627305    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n85kg\" (UniqueName: \"kubernetes.io/projected/7dd8a2ed-f55f-409c-956b-90085f39a31c-kube-api-access-n85kg\") pod \"model-serving-api-86f7b4b499-2khfb\" (UID: \"7dd8a2ed-f55f-409c-956b-90085f39a31c\") " pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:04.629987 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.629954    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd8a2ed-f55f-409c-956b-90085f39a31c-tls-certs\") pod \"model-serving-api-86f7b4b499-2khfb\" (UID: \"7dd8a2ed-f55f-409c-956b-90085f39a31c\") " pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:04.635094 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.635073    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85kg\" (UniqueName: \"kubernetes.io/projected/7dd8a2ed-f55f-409c-956b-90085f39a31c-kube-api-access-n85kg\") pod \"model-serving-api-86f7b4b499-2khfb\" (UID: \"7dd8a2ed-f55f-409c-956b-90085f39a31c\") " pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:04.686870 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.686830    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:04.810354 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:04.810320    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-2khfb"]
May 26 11:42:04.813081 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:42:04.813057    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd8a2ed_f55f_409c_956b_90085f39a31c.slice/crio-a6af7e35efe011757302d96d2c2f04c573dbee2cbf411af16dd79a5864ea0785 WatchSource:0}: Error finding container a6af7e35efe011757302d96d2c2f04c573dbee2cbf411af16dd79a5864ea0785: Status 404 returned error can't find the container with id a6af7e35efe011757302d96d2c2f04c573dbee2cbf411af16dd79a5864ea0785
May 26 11:42:05.532752 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:05.532713    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-2khfb" event={"ID":"7dd8a2ed-f55f-409c-956b-90085f39a31c","Type":"ContainerStarted","Data":"a6af7e35efe011757302d96d2c2f04c573dbee2cbf411af16dd79a5864ea0785"}
May 26 11:42:07.539848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:07.539740    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-2khfb" event={"ID":"7dd8a2ed-f55f-409c-956b-90085f39a31c","Type":"ContainerStarted","Data":"0a7568eee31852b69f37d7cb77b2017084b97d151afde5086c13907e86e28bc8"}
May 26 11:42:07.539848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:07.539845    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:42:07.558448 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:07.558393    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-2khfb" podStartSLOduration=1.260687433 podStartE2EDuration="3.558383056s" podCreationTimestamp="2026-05-26 11:42:04 +0000 UTC" firstStartedPulling="2026-05-26 11:42:04.814746703 +0000 UTC m=+887.177696350" lastFinishedPulling="2026-05-26 11:42:07.112442323 +0000 UTC m=+889.475391973" observedRunningTime="2026-05-26 11:42:07.556280183 +0000 UTC m=+889.919229863" watchObservedRunningTime="2026-05-26 11:42:07.558383056 +0000 UTC m=+889.921332724"
May 26 11:42:18.151000 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:18.150972    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:42:18.152393 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:18.152370    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:42:18.546545 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:42:18.546520    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-2khfb"
May 26 11:43:06.666116 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.666085    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"]
May 26 11:43:06.669315 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.669294    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.671247 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.671228    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
May 26 11:43:06.671772 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.671753    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 11:43:06.671882 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.671773    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 11:43:06.671882 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.671761    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b5sxz\""
May 26 11:43:06.671882 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.671807    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
May 26 11:43:06.679919 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.679902    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"]
May 26 11:43:06.765692 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.765656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30215e97-7798-4bb0-b19c-08f606ebe8ce-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.765869 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.765699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h67wh\" (UniqueName: \"kubernetes.io/projected/30215e97-7798-4bb0-b19c-08f606ebe8ce-kube-api-access-h67wh\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.765869 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.765753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.765869 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.765852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30215e97-7798-4bb0-b19c-08f606ebe8ce-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.866157 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.866125    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h67wh\" (UniqueName: \"kubernetes.io/projected/30215e97-7798-4bb0-b19c-08f606ebe8ce-kube-api-access-h67wh\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.866157 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.866158    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.866368 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.866203    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30215e97-7798-4bb0-b19c-08f606ebe8ce-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.866368 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.866228    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30215e97-7798-4bb0-b19c-08f606ebe8ce-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.866368 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:43:06.866353    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
May 26 11:43:06.866507 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:43:06.866419    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls podName:30215e97-7798-4bb0-b19c-08f606ebe8ce nodeName:}" failed. No retries permitted until 2026-05-26 11:43:07.366400109 +0000 UTC m=+949.729349758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls") pod "isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" (UID: "30215e97-7798-4bb0-b19c-08f606ebe8ce") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
May 26 11:43:06.866645 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.866623    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30215e97-7798-4bb0-b19c-08f606ebe8ce-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.866908 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.866892    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30215e97-7798-4bb0-b19c-08f606ebe8ce-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:06.877647 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:06.877616    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h67wh\" (UniqueName: \"kubernetes.io/projected/30215e97-7798-4bb0-b19c-08f606ebe8ce-kube-api-access-h67wh\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:07.371285 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:07.371243    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:07.373740 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:07.373714    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-ddg8z\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:07.579593 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:07.579545    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:07.707009 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:07.706985    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"]
May 26 11:43:07.711889 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:43:07.709750    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30215e97_7798_4bb0_b19c_08f606ebe8ce.slice/crio-3e8dcff608f6e2327ce12223a35a879f6effd260ff891f1fcb1ea850c40d5147 WatchSource:0}: Error finding container 3e8dcff608f6e2327ce12223a35a879f6effd260ff891f1fcb1ea850c40d5147: Status 404 returned error can't find the container with id 3e8dcff608f6e2327ce12223a35a879f6effd260ff891f1fcb1ea850c40d5147
May 26 11:43:08.700629 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:08.700588    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerStarted","Data":"3e8dcff608f6e2327ce12223a35a879f6effd260ff891f1fcb1ea850c40d5147"}
May 26 11:43:10.706973 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:10.706875    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerStarted","Data":"9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc"}
May 26 11:43:14.719194 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:14.719160    2563 generic.go:358] "Generic (PLEG): container finished" podID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerID="9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc" exitCode=0
May 26 11:43:14.719194 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:14.719197    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerDied","Data":"9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc"}
May 26 11:43:26.758389 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:26.758357    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerStarted","Data":"cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659"}
May 26 11:43:29.768816 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:29.768770    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerStarted","Data":"1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4"}
May 26 11:43:32.779181 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:32.779146    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerStarted","Data":"e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083"}
May 26 11:43:32.779577 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:32.779370    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:32.779577 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:32.779490    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:32.780606 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:32.780582    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:43:32.797566 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:32.797525    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podStartSLOduration=2.594608919 podStartE2EDuration="26.797514704s" podCreationTimestamp="2026-05-26 11:43:06 +0000 UTC" firstStartedPulling="2026-05-26 11:43:07.714544438 +0000 UTC m=+950.077494090" lastFinishedPulling="2026-05-26 11:43:31.917450211 +0000 UTC m=+974.280399875" observedRunningTime="2026-05-26 11:43:32.796289742 +0000 UTC m=+975.159239410" watchObservedRunningTime="2026-05-26 11:43:32.797514704 +0000 UTC m=+975.160464371"
May 26 11:43:33.781776 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:33.781738    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:43:33.782213 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:33.781884    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:33.782693 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:33.782655    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:43:33.785645 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:33.785619    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:43:34.784949 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:34.784911    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:43:34.785360 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:34.785242    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:43:35.787394 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:35.787304    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:43:35.787782 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:35.787602    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:43:45.787726 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:45.787683    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:43:45.788210 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:45.788102    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:43:55.787624 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:55.787566    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:43:55.788109 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:43:55.788087    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:05.787660 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:05.787603    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:44:05.788303 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:05.788166    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:15.787873 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:15.787827    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:44:15.788337 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:15.788283    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:25.787976 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:25.787932    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:44:25.788500 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:25.788477    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:35.788020 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:35.787983    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:44:35.788511 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:35.788482    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:44:41.721098 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.721060    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"]
May 26 11:44:41.721589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.721407    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" containerID="cri-o://e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083" gracePeriod=30
May 26 11:44:41.721589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.721379    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" containerID="cri-o://cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659" gracePeriod=30
May 26 11:44:41.721589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.721457    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy" containerID="cri-o://1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4" gracePeriod=30
May 26 11:44:41.881948 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.881915    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"]
May 26 11:44:41.885044 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.885027    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:41.887869 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.887849    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
May 26 11:44:41.894270 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.894252    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
May 26 11:44:41.920453 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.920429    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"]
May 26 11:44:41.965138 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.965105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e6283c9-68d8-4437-ba5e-1239b43130c1-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:41.965320 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.965146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e6283c9-68d8-4437-ba5e-1239b43130c1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:41.965320 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.965174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfw4m\" (UniqueName: \"kubernetes.io/projected/1e6283c9-68d8-4437-ba5e-1239b43130c1-kube-api-access-bfw4m\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:41.965320 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.965191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e6283c9-68d8-4437-ba5e-1239b43130c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:41.969623 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.969589    2563 generic.go:358] "Generic (PLEG): container finished" podID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerID="1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4" exitCode=2
May 26 11:44:41.969753 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:41.969661    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerDied","Data":"1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4"}
May 26 11:44:42.065951 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.065917    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e6283c9-68d8-4437-ba5e-1239b43130c1-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.066115 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.065955    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e6283c9-68d8-4437-ba5e-1239b43130c1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.066115 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.066014    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bfw4m\" (UniqueName: \"kubernetes.io/projected/1e6283c9-68d8-4437-ba5e-1239b43130c1-kube-api-access-bfw4m\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.066115 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.066046    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e6283c9-68d8-4437-ba5e-1239b43130c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.066477 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.066458    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e6283c9-68d8-4437-ba5e-1239b43130c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.066755 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.066736    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e6283c9-68d8-4437-ba5e-1239b43130c1-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.068656 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.068638    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e6283c9-68d8-4437-ba5e-1239b43130c1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.074285 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.074265    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfw4m\" (UniqueName: \"kubernetes.io/projected/1e6283c9-68d8-4437-ba5e-1239b43130c1-kube-api-access-bfw4m\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.194112 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.194077    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:42.321131 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.321041    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"]
May 26 11:44:42.324391 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:44:42.324340    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6283c9_68d8_4437_ba5e_1239b43130c1.slice/crio-02fe79419d2aa0afddd3391031263c04f9d444e68d054d8ce0453f33b0fdb7b6 WatchSource:0}: Error finding container 02fe79419d2aa0afddd3391031263c04f9d444e68d054d8ce0453f33b0fdb7b6: Status 404 returned error can't find the container with id 02fe79419d2aa0afddd3391031263c04f9d444e68d054d8ce0453f33b0fdb7b6
May 26 11:44:42.973679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.973640    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerStarted","Data":"cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d"}
May 26 11:44:42.973679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:42.973673    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerStarted","Data":"02fe79419d2aa0afddd3391031263c04f9d444e68d054d8ce0453f33b0fdb7b6"}
May 26 11:44:43.782301 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:43.782254    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
May 26 11:44:45.787272 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:45.787233    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:44:45.788879 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:45.788854    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:45.985597 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:45.985496    2563 generic.go:358] "Generic (PLEG): container finished" podID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerID="cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659" exitCode=0
May 26 11:44:45.985597 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:45.985551    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerDied","Data":"cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659"}
May 26 11:44:46.989709 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:46.989673    2563 generic.go:358] "Generic (PLEG): container finished" podID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerID="cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d" exitCode=0
May 26 11:44:46.990080 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:46.989747    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerDied","Data":"cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d"}
May 26 11:44:47.994075 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:47.994037    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerStarted","Data":"35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771"}
May 26 11:44:47.994448 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:47.994083    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerStarted","Data":"08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184"}
May 26 11:44:47.994448 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:47.994096    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerStarted","Data":"afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58"}
May 26 11:44:47.994448 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:47.994397    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:47.994585 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:47.994528    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:47.995727 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:47.995701    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:44:48.013550 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:48.013511    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podStartSLOduration=7.013500278 podStartE2EDuration="7.013500278s" podCreationTimestamp="2026-05-26 11:44:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:44:48.01202027 +0000 UTC m=+1050.374969939" watchObservedRunningTime="2026-05-26 11:44:48.013500278 +0000 UTC m=+1050.376449944"
May 26 11:44:48.782578 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:48.782536    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
May 26 11:44:48.996589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:48.996558    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:48.997023 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:48.996627    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:44:48.997697 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:48.997675    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:49.999870 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:49.999829    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:44:50.000289 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:50.000092    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:53.782493 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:53.782452    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
May 26 11:44:53.784871 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:53.782572    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:44:55.003418 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:55.003385    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:44:55.003958 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:55.003926    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:44:55.004257 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:55.004217    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:55.787508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:55.787468    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:44:55.789029 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:55.789005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:44:58.782379 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:44:58.782336    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
May 26 11:45:03.783089 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:03.783044    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
May 26 11:45:05.004439 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:05.004400    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:45:05.004936 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:05.004867    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:05.787506 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:05.787419    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
May 26 11:45:05.787663 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:05.787541    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:45:05.789079 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:05.789050    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:05.789200 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:05.789161    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:45:08.782216 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:08.782172    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
May 26 11:45:11.870243 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.870219    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:45:11.996735 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.996644    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h67wh\" (UniqueName: \"kubernetes.io/projected/30215e97-7798-4bb0-b19c-08f606ebe8ce-kube-api-access-h67wh\") pod \"30215e97-7798-4bb0-b19c-08f606ebe8ce\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") "
May 26 11:45:11.996735 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.996709    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30215e97-7798-4bb0-b19c-08f606ebe8ce-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"30215e97-7798-4bb0-b19c-08f606ebe8ce\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") "
May 26 11:45:11.996735 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.996728    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls\") pod \"30215e97-7798-4bb0-b19c-08f606ebe8ce\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") "
May 26 11:45:11.996980 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.996758    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30215e97-7798-4bb0-b19c-08f606ebe8ce-kserve-provision-location\") pod \"30215e97-7798-4bb0-b19c-08f606ebe8ce\" (UID: \"30215e97-7798-4bb0-b19c-08f606ebe8ce\") "
May 26 11:45:11.997120 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.997100    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30215e97-7798-4bb0-b19c-08f606ebe8ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30215e97-7798-4bb0-b19c-08f606ebe8ce" (UID: "30215e97-7798-4bb0-b19c-08f606ebe8ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:11.997164 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.997103    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30215e97-7798-4bb0-b19c-08f606ebe8ce-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "30215e97-7798-4bb0-b19c-08f606ebe8ce" (UID: "30215e97-7798-4bb0-b19c-08f606ebe8ce"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:45:11.999168 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.999142    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30215e97-7798-4bb0-b19c-08f606ebe8ce-kube-api-access-h67wh" (OuterVolumeSpecName: "kube-api-access-h67wh") pod "30215e97-7798-4bb0-b19c-08f606ebe8ce" (UID: "30215e97-7798-4bb0-b19c-08f606ebe8ce"). InnerVolumeSpecName "kube-api-access-h67wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:45:11.999305 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:11.999228    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30215e97-7798-4bb0-b19c-08f606ebe8ce" (UID: "30215e97-7798-4bb0-b19c-08f606ebe8ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:45:12.060733 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.060699    2563 generic.go:358] "Generic (PLEG): container finished" podID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerID="e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083" exitCode=0
May 26 11:45:12.060915 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.060786    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"
May 26 11:45:12.060915 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.060788    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerDied","Data":"e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083"}
May 26 11:45:12.060915 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.060910    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z" event={"ID":"30215e97-7798-4bb0-b19c-08f606ebe8ce","Type":"ContainerDied","Data":"3e8dcff608f6e2327ce12223a35a879f6effd260ff891f1fcb1ea850c40d5147"}
May 26 11:45:12.061050 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.060926    2563 scope.go:117] "RemoveContainer" containerID="e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083"
May 26 11:45:12.068764 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.068739    2563 scope.go:117] "RemoveContainer" containerID="1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4"
May 26 11:45:12.075874 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.075855    2563 scope.go:117] "RemoveContainer" containerID="cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659"
May 26 11:45:12.081880 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.081858    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"]
May 26 11:45:12.083747 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.083731    2563 scope.go:117] "RemoveContainer" containerID="9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc"
May 26 11:45:12.086681 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.086659    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-ddg8z"]
May 26 11:45:12.090679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.090661    2563 scope.go:117] "RemoveContainer" containerID="e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083"
May 26 11:45:12.090969 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:45:12.090950    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083\": container with ID starting with e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083 not found: ID does not exist" containerID="e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083"
May 26 11:45:12.091026 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.090978    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083"} err="failed to get container status \"e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083\": rpc error: code = NotFound desc = could not find container \"e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083\": container with ID starting with e0437fe2cafd531d68332825cf0584c0b346f7e451630cc9baf66f8000498083 not found: ID does not exist"
May 26 11:45:12.091026 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.090998    2563 scope.go:117] "RemoveContainer" containerID="1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4"
May 26 11:45:12.091238 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:45:12.091218    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4\": container with ID starting with 1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4 not found: ID does not exist" containerID="1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4"
May 26 11:45:12.091288 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.091245    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4"} err="failed to get container status \"1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4\": rpc error: code = NotFound desc = could not find container \"1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4\": container with ID starting with 1c40cb7fecf8e25968bc20e07ff8c77867cc898aee1a83380589c6eec6f66fa4 not found: ID does not exist"
May 26 11:45:12.091288 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.091270    2563 scope.go:117] "RemoveContainer" containerID="cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659"
May 26 11:45:12.091498 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:45:12.091484    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659\": container with ID starting with cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659 not found: ID does not exist" containerID="cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659"
May 26 11:45:12.091537 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.091500    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659"} err="failed to get container status \"cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659\": rpc error: code = NotFound desc = could not find container \"cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659\": container with ID starting with cc6f27c06002678f6aa679e02f3d28be01f923a8d3d561b81166349c396f2659 not found: ID does not exist"
May 26 11:45:12.091537 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.091514    2563 scope.go:117] "RemoveContainer" containerID="9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc"
May 26 11:45:12.091711 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:45:12.091696    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc\": container with ID starting with 9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc not found: ID does not exist" containerID="9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc"
May 26 11:45:12.091750 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.091718    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc"} err="failed to get container status \"9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc\": rpc error: code = NotFound desc = could not find container \"9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc\": container with ID starting with 9bd30a7801d3438224714e9dc3dacd1d785ad82aee805ba5a8d9c4e9811fa4bc not found: ID does not exist"
May 26 11:45:12.097592 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.097576    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h67wh\" (UniqueName: \"kubernetes.io/projected/30215e97-7798-4bb0-b19c-08f606ebe8ce-kube-api-access-h67wh\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:45:12.097661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.097596    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30215e97-7798-4bb0-b19c-08f606ebe8ce-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:45:12.097661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.097606    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30215e97-7798-4bb0-b19c-08f606ebe8ce-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:45:12.097661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.097615    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30215e97-7798-4bb0-b19c-08f606ebe8ce-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:45:12.241947 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:12.241916    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" path="/var/lib/kubelet/pods/30215e97-7798-4bb0-b19c-08f606ebe8ce/volumes"
May 26 11:45:15.004135 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:15.004086    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:45:15.004595 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:15.004474    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:25.004710 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:25.004662    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:45:25.005144 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:25.005058    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:35.003949 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:35.003910    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:45:35.004419 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:35.004258    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:45.004764 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:45.004726    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:45:45.005289 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:45.005219    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:45:55.004478 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:55.004444    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:45:55.005007 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:45:55.004874    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:46:07.014559 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.014522    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"]
May 26 11:46:07.014985 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.014879    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" containerID="cri-o://afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58" gracePeriod=30
May 26 11:46:07.014985 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.014918    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" containerID="cri-o://35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771" gracePeriod=30
May 26 11:46:07.014985 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.014918    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy" containerID="cri-o://08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184" gracePeriod=30
May 26 11:46:07.104537 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104505    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"]
May 26 11:46:07.104813 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104785    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="storage-initializer"
May 26 11:46:07.104813 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104814    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="storage-initializer"
May 26 11:46:07.104904 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104822    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy"
May 26 11:46:07.104904 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104828    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy"
May 26 11:46:07.104904 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104838    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent"
May 26 11:46:07.104904 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104844    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent"
May 26 11:46:07.104904 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104855    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container"
May 26 11:46:07.104904 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104861    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container"
May 26 11:46:07.104904 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104906    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kserve-container"
May 26 11:46:07.105097 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104915    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="kube-rbac-proxy"
May 26 11:46:07.105097 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.104923    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="30215e97-7798-4bb0-b19c-08f606ebe8ce" containerName="agent"
May 26 11:46:07.107935 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.107919    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.111572 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.111552    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
May 26 11:46:07.111670 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.111632    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
May 26 11:46:07.121515 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.121487    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"]
May 26 11:46:07.188235 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.188196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44859f60-6eed-49bd-9550-ca2e825cd47b-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.188413 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.188251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmr7c\" (UniqueName: \"kubernetes.io/projected/44859f60-6eed-49bd-9550-ca2e825cd47b-kube-api-access-gmr7c\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.188413 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.188278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44859f60-6eed-49bd-9550-ca2e825cd47b-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.214309 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.214223    2563 generic.go:358] "Generic (PLEG): container finished" podID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerID="08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184" exitCode=2
May 26 11:46:07.214309 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.214270    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerDied","Data":"08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184"}
May 26 11:46:07.289486 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.289396    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gmr7c\" (UniqueName: \"kubernetes.io/projected/44859f60-6eed-49bd-9550-ca2e825cd47b-kube-api-access-gmr7c\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.289486 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.289436    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44859f60-6eed-49bd-9550-ca2e825cd47b-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.289681 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.289574    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44859f60-6eed-49bd-9550-ca2e825cd47b-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.290211 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.290189    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44859f60-6eed-49bd-9550-ca2e825cd47b-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.291965 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.291949    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44859f60-6eed-49bd-9550-ca2e825cd47b-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.297861 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.297839    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmr7c\" (UniqueName: \"kubernetes.io/projected/44859f60-6eed-49bd-9550-ca2e825cd47b-kube-api-access-gmr7c\") pod \"message-dumper-predictor-c7d86bcbd-b9npv\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.418119 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.418081    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:07.544227 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.544198    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"]
May 26 11:46:07.547107 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:46:07.547081    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44859f60_6eed_49bd_9550_ca2e825cd47b.slice/crio-68f61051ff5ed49058aee23a1ee9a838ba7a3cc3641434c362156d87d05146aa WatchSource:0}: Error finding container 68f61051ff5ed49058aee23a1ee9a838ba7a3cc3641434c362156d87d05146aa: Status 404 returned error can't find the container with id 68f61051ff5ed49058aee23a1ee9a838ba7a3cc3641434c362156d87d05146aa
May 26 11:46:07.549067 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:07.549050    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:46:08.218395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:08.218361    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" event={"ID":"44859f60-6eed-49bd-9550-ca2e825cd47b","Type":"ContainerStarted","Data":"68f61051ff5ed49058aee23a1ee9a838ba7a3cc3641434c362156d87d05146aa"}
May 26 11:46:09.222223 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:09.222178    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" event={"ID":"44859f60-6eed-49bd-9550-ca2e825cd47b","Type":"ContainerStarted","Data":"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7"}
May 26 11:46:09.222223 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:09.222227    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" event={"ID":"44859f60-6eed-49bd-9550-ca2e825cd47b","Type":"ContainerStarted","Data":"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01"}
May 26 11:46:09.222620 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:09.222373    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:09.257725 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:09.257654    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" podStartSLOduration=1.240360111 podStartE2EDuration="2.25763312s" podCreationTimestamp="2026-05-26 11:46:07 +0000 UTC" firstStartedPulling="2026-05-26 11:46:07.549208866 +0000 UTC m=+1129.912158513" lastFinishedPulling="2026-05-26 11:46:08.566481876 +0000 UTC m=+1130.929431522" observedRunningTime="2026-05-26 11:46:09.256680449 +0000 UTC m=+1131.619630119" watchObservedRunningTime="2026-05-26 11:46:09.25763312 +0000 UTC m=+1131.620582789"
May 26 11:46:10.000527 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:10.000479    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:46:10.224976 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:10.224941    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:10.226362 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:10.226339    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:12.233087 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:12.233052    2563 generic.go:358] "Generic (PLEG): container finished" podID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerID="afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58" exitCode=0
May 26 11:46:12.233446 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:12.233126    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerDied","Data":"afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58"}
May 26 11:46:15.000575 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:15.000521    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:46:15.004896 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:15.004862    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:46:15.005152 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:15.005128    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:46:17.237391 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:17.237361    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:46:20.000663 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:20.000619    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:46:20.001056 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:20.000754    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:46:25.000672 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:25.000627    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:46:25.004390 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:25.004359    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:46:25.005548 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:25.005521    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:46:27.061035 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.061000    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"]
May 26 11:46:27.064579 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.064560    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.066519 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.066498    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
May 26 11:46:27.066683 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.066587    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
May 26 11:46:27.074064 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.074038    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"]
May 26 11:46:27.135637 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.135601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.135637 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.135639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.135878 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.135658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtg56\" (UniqueName: \"kubernetes.io/projected/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kube-api-access-mtg56\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.135878 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.135752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.236498 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.236455    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.236498 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.236499    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg56\" (UniqueName: \"kubernetes.io/projected/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kube-api-access-mtg56\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.236735 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.236532    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.236735 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.236603    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.237060 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.237035    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.237435 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.237406    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.239114 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.239096    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.245073 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.245047    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtg56\" (UniqueName: \"kubernetes.io/projected/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kube-api-access-mtg56\") pod \"isvc-logger-predictor-6d94fc7bdd-g944p\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.375174 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.375086    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:27.502983 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:27.502952    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"]
May 26 11:46:27.506652 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:46:27.506618    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdfc4d2_ce64_4960_9cf9_bfbd99fc7266.slice/crio-39f6a818b9163bae13b9b0a4ae6f002973df970a44a68d1be7e4a3244f82acef WatchSource:0}: Error finding container 39f6a818b9163bae13b9b0a4ae6f002973df970a44a68d1be7e4a3244f82acef: Status 404 returned error can't find the container with id 39f6a818b9163bae13b9b0a4ae6f002973df970a44a68d1be7e4a3244f82acef
May 26 11:46:28.278472 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:28.278435    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerStarted","Data":"81e44783406d57fb58b62455f42de0e00409d4707a6445ca74acc5a9108b257d"}
May 26 11:46:28.278472 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:28.278477    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerStarted","Data":"39f6a818b9163bae13b9b0a4ae6f002973df970a44a68d1be7e4a3244f82acef"}
May 26 11:46:30.000490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:30.000446    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:46:31.289004 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:31.288966    2563 generic.go:358] "Generic (PLEG): container finished" podID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerID="81e44783406d57fb58b62455f42de0e00409d4707a6445ca74acc5a9108b257d" exitCode=0
May 26 11:46:31.289369 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:31.289025    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerDied","Data":"81e44783406d57fb58b62455f42de0e00409d4707a6445ca74acc5a9108b257d"}
May 26 11:46:32.294063 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:32.294026    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerStarted","Data":"dcdc05ae397ee3b32bc3f51d0095184064ac5523b7e236faba14bb324360e0ed"}
May 26 11:46:32.294063 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:32.294068    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerStarted","Data":"9fb251df325ede3d7e561f42d2f7bfce3434ca55de33bc03b5fa319bdd6df7b4"}
May 26 11:46:32.294589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:32.294081    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerStarted","Data":"08b44ac84a1c8023ad34b6646e8cb862d82068380ffb7800abe8a398eec7c6e7"}
May 26 11:46:32.294589 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:32.294332    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:32.313204 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:32.313160    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podStartSLOduration=5.313146542 podStartE2EDuration="5.313146542s" podCreationTimestamp="2026-05-26 11:46:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:46:32.312097899 +0000 UTC m=+1154.675047568" watchObservedRunningTime="2026-05-26 11:46:32.313146542 +0000 UTC m=+1154.676096211"
May 26 11:46:33.297425 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:33.297399    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:33.297826 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:33.297441    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:33.298756 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:33.298719    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:46:33.299365 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:33.299341    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:46:34.300151 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:34.300105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:46:34.300586 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:34.300563    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:46:35.000358 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:35.000314    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:46:35.004332 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:35.004298    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:46:35.004439 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:35.004423    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:46:35.005494 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:35.005467    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:5000: connect: connection refused"
May 26 11:46:35.005572 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:35.005560    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:46:37.172074 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.172052    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:46:37.214879 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.214843    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e6283c9-68d8-4437-ba5e-1239b43130c1-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"1e6283c9-68d8-4437-ba5e-1239b43130c1\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") "
May 26 11:46:37.215035 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.214918    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e6283c9-68d8-4437-ba5e-1239b43130c1-kserve-provision-location\") pod \"1e6283c9-68d8-4437-ba5e-1239b43130c1\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") "
May 26 11:46:37.215035 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.215003    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfw4m\" (UniqueName: \"kubernetes.io/projected/1e6283c9-68d8-4437-ba5e-1239b43130c1-kube-api-access-bfw4m\") pod \"1e6283c9-68d8-4437-ba5e-1239b43130c1\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") "
May 26 11:46:37.215120 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.215047    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e6283c9-68d8-4437-ba5e-1239b43130c1-proxy-tls\") pod \"1e6283c9-68d8-4437-ba5e-1239b43130c1\" (UID: \"1e6283c9-68d8-4437-ba5e-1239b43130c1\") "
May 26 11:46:37.215232 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.215210    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6283c9-68d8-4437-ba5e-1239b43130c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e6283c9-68d8-4437-ba5e-1239b43130c1" (UID: "1e6283c9-68d8-4437-ba5e-1239b43130c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:46:37.215299 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.215213    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6283c9-68d8-4437-ba5e-1239b43130c1-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "1e6283c9-68d8-4437-ba5e-1239b43130c1" (UID: "1e6283c9-68d8-4437-ba5e-1239b43130c1"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:46:37.215299 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.215275    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e6283c9-68d8-4437-ba5e-1239b43130c1-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:46:37.215299 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.215288    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e6283c9-68d8-4437-ba5e-1239b43130c1-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:46:37.217113 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.217090    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6283c9-68d8-4437-ba5e-1239b43130c1-kube-api-access-bfw4m" (OuterVolumeSpecName: "kube-api-access-bfw4m") pod "1e6283c9-68d8-4437-ba5e-1239b43130c1" (UID: "1e6283c9-68d8-4437-ba5e-1239b43130c1"). InnerVolumeSpecName "kube-api-access-bfw4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:46:37.217211 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.217160    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6283c9-68d8-4437-ba5e-1239b43130c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e6283c9-68d8-4437-ba5e-1239b43130c1" (UID: "1e6283c9-68d8-4437-ba5e-1239b43130c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:46:37.311201 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.311172    2563 generic.go:358] "Generic (PLEG): container finished" podID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerID="35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771" exitCode=0
May 26 11:46:37.311346 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.311236    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerDied","Data":"35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771"}
May 26 11:46:37.311346 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.311263    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s" event={"ID":"1e6283c9-68d8-4437-ba5e-1239b43130c1","Type":"ContainerDied","Data":"02fe79419d2aa0afddd3391031263c04f9d444e68d054d8ce0453f33b0fdb7b6"}
May 26 11:46:37.311346 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.311279    2563 scope.go:117] "RemoveContainer" containerID="35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771"
May 26 11:46:37.311346 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.311325    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"
May 26 11:46:37.315818 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.315777    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfw4m\" (UniqueName: \"kubernetes.io/projected/1e6283c9-68d8-4437-ba5e-1239b43130c1-kube-api-access-bfw4m\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:46:37.315927 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.315823    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e6283c9-68d8-4437-ba5e-1239b43130c1-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:46:37.319539 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.319520    2563 scope.go:117] "RemoveContainer" containerID="08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184"
May 26 11:46:37.326601 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.326583    2563 scope.go:117] "RemoveContainer" containerID="afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58"
May 26 11:46:37.332854 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.332816    2563 scope.go:117] "RemoveContainer" containerID="cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d"
May 26 11:46:37.335537 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.335518    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"]
May 26 11:46:37.339880 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.339857    2563 scope.go:117] "RemoveContainer" containerID="35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771"
May 26 11:46:37.340232 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:46:37.340196    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771\": container with ID starting with 35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771 not found: ID does not exist" containerID="35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771"
May 26 11:46:37.340388 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.340240    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771"} err="failed to get container status \"35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771\": rpc error: code = NotFound desc = could not find container \"35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771\": container with ID starting with 35791c97811ff7dea7cc48ec5c4bbaf948cfbaffefa9f450e29955252538e771 not found: ID does not exist"
May 26 11:46:37.340388 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.340257    2563 scope.go:117] "RemoveContainer" containerID="08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184"
May 26 11:46:37.340523 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:46:37.340507    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184\": container with ID starting with 08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184 not found: ID does not exist" containerID="08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184"
May 26 11:46:37.340562 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.340530    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184"} err="failed to get container status \"08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184\": rpc error: code = NotFound desc = could not find container \"08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184\": container with ID starting with 08011b413e54ca1d18e13b185bbca10101896c084ce1c369ab0ca378e5e2b184 not found: ID does not exist"
May 26 11:46:37.340562 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.340548    2563 scope.go:117] "RemoveContainer" containerID="afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58"
May 26 11:46:37.340767 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:46:37.340751    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58\": container with ID starting with afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58 not found: ID does not exist" containerID="afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58"
May 26 11:46:37.340844 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.340772    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58"} err="failed to get container status \"afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58\": rpc error: code = NotFound desc = could not find container \"afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58\": container with ID starting with afbd208d273e6a0ae94bcd09db8036f2e604bd2434800b53109789bf9fae7b58 not found: ID does not exist"
May 26 11:46:37.340844 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.340787    2563 scope.go:117] "RemoveContainer" containerID="cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d"
May 26 11:46:37.341068 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:46:37.341052    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d\": container with ID starting with cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d not found: ID does not exist" containerID="cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d"
May 26 11:46:37.341113 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.341071    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d"} err="failed to get container status \"cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d\": rpc error: code = NotFound desc = could not find container \"cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d\": container with ID starting with cde90aa8b12ce3c84f54c9c50ae0aeaeb195f0709c232aeb5b1d71420aa9ab0d not found: ID does not exist"
May 26 11:46:37.341383 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:37.341365    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-stx2s"]
May 26 11:46:38.242442 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:38.242407    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" path="/var/lib/kubelet/pods/1e6283c9-68d8-4437-ba5e-1239b43130c1/volumes"
May 26 11:46:39.305325 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:39.305295    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:46:39.305914 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:39.305888    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:46:39.306239 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:39.306218    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:46:49.305997 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:49.305947    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:46:49.306441 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:49.306387    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:46:59.305856 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:59.305809    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:46:59.306420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:46:59.306265    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:47:09.306562 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:09.306519    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:47:09.307134 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:09.307111    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:47:18.169654 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:18.169626    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:47:18.171214 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:18.171194    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:47:19.306403 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:19.306362    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:47:19.306858 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:19.306682    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:47:29.306202 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:29.306157    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:47:29.306679 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:29.306653    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:47:39.307035 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:39.307005    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:47:39.307497 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:39.307072    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:47:52.080097 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.080063    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-b9npv_44859f60-6eed-49bd-9550-ca2e825cd47b/kserve-container/0.log"
May 26 11:47:52.301095 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.301062    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"]
May 26 11:47:52.301483 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.301442    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" containerID="cri-o://08b44ac84a1c8023ad34b6646e8cb862d82068380ffb7800abe8a398eec7c6e7" gracePeriod=30
May 26 11:47:52.301615 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.301495    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" containerID="cri-o://dcdc05ae397ee3b32bc3f51d0095184064ac5523b7e236faba14bb324360e0ed" gracePeriod=30
May 26 11:47:52.301615 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.301541    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy" containerID="cri-o://9fb251df325ede3d7e561f42d2f7bfce3434ca55de33bc03b5fa319bdd6df7b4" gracePeriod=30
May 26 11:47:52.362972 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.362911    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"]
May 26 11:47:52.363210 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363197    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="storage-initializer"
May 26 11:47:52.363251 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363211    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="storage-initializer"
May 26 11:47:52.363251 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363221    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent"
May 26 11:47:52.363251 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363226    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent"
May 26 11:47:52.363251 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363235    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container"
May 26 11:47:52.363251 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363241    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container"
May 26 11:47:52.363251 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363249    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy"
May 26 11:47:52.363420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363254    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy"
May 26 11:47:52.363420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363311    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kube-rbac-proxy"
May 26 11:47:52.363420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363320    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="agent"
May 26 11:47:52.363420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.363327    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="1e6283c9-68d8-4437-ba5e-1239b43130c1" containerName="kserve-container"
May 26 11:47:52.366101 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.366083    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.368721 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.368703    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
May 26 11:47:52.369023 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.369009    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
May 26 11:47:52.379909 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.379883    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"]
May 26 11:47:52.418057 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.418017    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"]
May 26 11:47:52.418371 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.418344    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerName="kserve-container" containerID="cri-o://886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01" gracePeriod=30
May 26 11:47:52.418479 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.418385    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerName="kube-rbac-proxy" containerID="cri-o://fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7" gracePeriod=30
May 26 11:47:52.460063 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.460036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.460181 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.460070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9j9\" (UniqueName: \"kubernetes.io/projected/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kube-api-access-ph9j9\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.460181 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.460089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.460279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.460172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.518130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.518099    2563 generic.go:358] "Generic (PLEG): container finished" podID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerID="9fb251df325ede3d7e561f42d2f7bfce3434ca55de33bc03b5fa319bdd6df7b4" exitCode=2
May 26 11:47:52.518248 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.518152    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerDied","Data":"9fb251df325ede3d7e561f42d2f7bfce3434ca55de33bc03b5fa319bdd6df7b4"}
May 26 11:47:52.561050 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.561007    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.561222 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.561108    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.561222 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.561141    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9j9\" (UniqueName: \"kubernetes.io/projected/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kube-api-access-ph9j9\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.561222 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.561169    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.562119 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.561443    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.562119 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.561897    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.564231 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.564211    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.570076 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.570056    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9j9\" (UniqueName: \"kubernetes.io/projected/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kube-api-access-ph9j9\") pod \"isvc-lightgbm-predictor-bdf964bd-mwn72\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.656542 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.656519    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:47:52.661512 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.661493    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44859f60-6eed-49bd-9550-ca2e825cd47b-message-dumper-kube-rbac-proxy-sar-config\") pod \"44859f60-6eed-49bd-9550-ca2e825cd47b\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") "
May 26 11:47:52.661603 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.661542    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmr7c\" (UniqueName: \"kubernetes.io/projected/44859f60-6eed-49bd-9550-ca2e825cd47b-kube-api-access-gmr7c\") pod \"44859f60-6eed-49bd-9550-ca2e825cd47b\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") "
May 26 11:47:52.661603 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.661576    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44859f60-6eed-49bd-9550-ca2e825cd47b-proxy-tls\") pod \"44859f60-6eed-49bd-9550-ca2e825cd47b\" (UID: \"44859f60-6eed-49bd-9550-ca2e825cd47b\") "
May 26 11:47:52.661821 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.661778    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44859f60-6eed-49bd-9550-ca2e825cd47b-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "44859f60-6eed-49bd-9550-ca2e825cd47b" (UID: "44859f60-6eed-49bd-9550-ca2e825cd47b"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:47:52.663709 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.663686    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44859f60-6eed-49bd-9550-ca2e825cd47b-kube-api-access-gmr7c" (OuterVolumeSpecName: "kube-api-access-gmr7c") pod "44859f60-6eed-49bd-9550-ca2e825cd47b" (UID: "44859f60-6eed-49bd-9550-ca2e825cd47b"). InnerVolumeSpecName "kube-api-access-gmr7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:47:52.663804 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.663704    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44859f60-6eed-49bd-9550-ca2e825cd47b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44859f60-6eed-49bd-9550-ca2e825cd47b" (UID: "44859f60-6eed-49bd-9550-ca2e825cd47b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:47:52.679654 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.679635    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:47:52.762481 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.762448    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmr7c\" (UniqueName: \"kubernetes.io/projected/44859f60-6eed-49bd-9550-ca2e825cd47b-kube-api-access-gmr7c\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:47:52.762601 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.762488    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44859f60-6eed-49bd-9550-ca2e825cd47b-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:47:52.762601 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.762506    2563 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44859f60-6eed-49bd-9550-ca2e825cd47b-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:47:52.808077 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:52.808044    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"]
May 26 11:47:52.811088 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:47:52.811059    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode397c5f5_7263_4bc7_b38f_2ec61b4b9596.slice/crio-51a8e855d23886ea75cfade8d5214ae7aa132227ff43cf66367c95afc156cdda WatchSource:0}: Error finding container 51a8e855d23886ea75cfade8d5214ae7aa132227ff43cf66367c95afc156cdda: Status 404 returned error can't find the container with id 51a8e855d23886ea75cfade8d5214ae7aa132227ff43cf66367c95afc156cdda
May 26 11:47:53.521694 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.521657    2563 generic.go:358] "Generic (PLEG): container finished" podID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerID="fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7" exitCode=2
May 26 11:47:53.521694 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.521691    2563 generic.go:358] "Generic (PLEG): container finished" podID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerID="886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01" exitCode=2
May 26 11:47:53.522176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.521742    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"
May 26 11:47:53.522176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.521738    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" event={"ID":"44859f60-6eed-49bd-9550-ca2e825cd47b","Type":"ContainerDied","Data":"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7"}
May 26 11:47:53.522176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.521833    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" event={"ID":"44859f60-6eed-49bd-9550-ca2e825cd47b","Type":"ContainerDied","Data":"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01"}
May 26 11:47:53.522176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.521850    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv" event={"ID":"44859f60-6eed-49bd-9550-ca2e825cd47b","Type":"ContainerDied","Data":"68f61051ff5ed49058aee23a1ee9a838ba7a3cc3641434c362156d87d05146aa"}
May 26 11:47:53.522176 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.521871    2563 scope.go:117] "RemoveContainer" containerID="fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7"
May 26 11:47:53.523115 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.523089    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" event={"ID":"e397c5f5-7263-4bc7-b38f-2ec61b4b9596","Type":"ContainerStarted","Data":"d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046"}
May 26 11:47:53.523178 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.523126    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" event={"ID":"e397c5f5-7263-4bc7-b38f-2ec61b4b9596","Type":"ContainerStarted","Data":"51a8e855d23886ea75cfade8d5214ae7aa132227ff43cf66367c95afc156cdda"}
May 26 11:47:53.529554 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.529440    2563 scope.go:117] "RemoveContainer" containerID="886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01"
May 26 11:47:53.536062 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.536046    2563 scope.go:117] "RemoveContainer" containerID="fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7"
May 26 11:47:53.536319 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:47:53.536301    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7\": container with ID starting with fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7 not found: ID does not exist" containerID="fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7"
May 26 11:47:53.536366 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.536327    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7"} err="failed to get container status \"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7\": rpc error: code = NotFound desc = could not find container \"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7\": container with ID starting with fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7 not found: ID does not exist"
May 26 11:47:53.536366 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.536345    2563 scope.go:117] "RemoveContainer" containerID="886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01"
May 26 11:47:53.536567 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:47:53.536548    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01\": container with ID starting with 886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01 not found: ID does not exist" containerID="886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01"
May 26 11:47:53.536610 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.536574    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01"} err="failed to get container status \"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01\": rpc error: code = NotFound desc = could not find container \"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01\": container with ID starting with 886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01 not found: ID does not exist"
May 26 11:47:53.536610 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.536590    2563 scope.go:117] "RemoveContainer" containerID="fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7"
May 26 11:47:53.536823 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.536783    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7"} err="failed to get container status \"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7\": rpc error: code = NotFound desc = could not find container \"fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7\": container with ID starting with fc95bb07d6a7e111128d450bf2e222cf164b32da10b7e9abc068cb7f18cec9e7 not found: ID does not exist"
May 26 11:47:53.536902 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.536827    2563 scope.go:117] "RemoveContainer" containerID="886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01"
May 26 11:47:53.537053 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.537031    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01"} err="failed to get container status \"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01\": rpc error: code = NotFound desc = could not find container \"886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01\": container with ID starting with 886317efbcaa9685e8b4aadd05472271c197e4ee441a319b5d7eeedff217bd01 not found: ID does not exist"
May 26 11:47:53.556443 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.556420    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"]
May 26 11:47:53.560252 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:53.560231    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-b9npv"]
May 26 11:47:54.242044 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:54.242010    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" path="/var/lib/kubelet/pods/44859f60-6eed-49bd-9550-ca2e825cd47b/volumes"
May 26 11:47:54.300844 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:54.300772    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 26 11:47:56.533010 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:56.532969    2563 generic.go:358] "Generic (PLEG): container finished" podID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerID="d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046" exitCode=0
May 26 11:47:56.533397 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:56.533042    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" event={"ID":"e397c5f5-7263-4bc7-b38f-2ec61b4b9596","Type":"ContainerDied","Data":"d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046"}
May 26 11:47:56.535289 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:56.535265    2563 generic.go:358] "Generic (PLEG): container finished" podID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerID="08b44ac84a1c8023ad34b6646e8cb862d82068380ffb7800abe8a398eec7c6e7" exitCode=0
May 26 11:47:56.535361 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:56.535319    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerDied","Data":"08b44ac84a1c8023ad34b6646e8cb862d82068380ffb7800abe8a398eec7c6e7"}
May 26 11:47:59.301259 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:59.301214    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 26 11:47:59.306667 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:59.306629    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:47:59.307019 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:47:59.306997    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:48:03.561250 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:03.561215    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" event={"ID":"e397c5f5-7263-4bc7-b38f-2ec61b4b9596","Type":"ContainerStarted","Data":"720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148"}
May 26 11:48:03.561250 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:03.561257    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" event={"ID":"e397c5f5-7263-4bc7-b38f-2ec61b4b9596","Type":"ContainerStarted","Data":"d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d"}
May 26 11:48:03.561651 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:03.561447    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:48:03.592352 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:03.592307    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podStartSLOduration=5.27150628 podStartE2EDuration="11.592293568s" podCreationTimestamp="2026-05-26 11:47:52 +0000 UTC" firstStartedPulling="2026-05-26 11:47:56.534484873 +0000 UTC m=+1238.897434520" lastFinishedPulling="2026-05-26 11:48:02.855272161 +0000 UTC m=+1245.218221808" observedRunningTime="2026-05-26 11:48:03.592128639 +0000 UTC m=+1245.955078309" watchObservedRunningTime="2026-05-26 11:48:03.592293568 +0000 UTC m=+1245.955243236"
May 26 11:48:04.300605 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:04.300554    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 26 11:48:04.300775 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:04.300686    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:48:04.564245 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:04.564158    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:48:04.565174 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:04.565141    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:48:05.566477 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:05.566438    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:48:09.300568 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:09.300523    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 26 11:48:09.305999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:09.305966    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:48:09.306309 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:09.306288    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:48:10.570690 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:10.570658    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:48:10.571399 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:10.571344    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:48:14.301219 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:14.301181    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 26 11:48:19.300777 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:19.300727    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 26 11:48:19.306118 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:19.306084    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:48:19.306273 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:19.306259    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:48:19.306467 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:19.306428    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:48:19.306560 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:19.306545    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:48:20.572130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:20.572089    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:48:22.615068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:22.614978    2563 generic.go:358] "Generic (PLEG): container finished" podID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerID="dcdc05ae397ee3b32bc3f51d0095184064ac5523b7e236faba14bb324360e0ed" exitCode=0
May 26 11:48:22.615068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:22.615045    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerDied","Data":"dcdc05ae397ee3b32bc3f51d0095184064ac5523b7e236faba14bb324360e0ed"}
May 26 11:48:22.950921 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:22.950896    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:48:23.086507 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.086464    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-proxy-tls\") pod \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") "
May 26 11:48:23.086507 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.086506    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtg56\" (UniqueName: \"kubernetes.io/projected/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kube-api-access-mtg56\") pod \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") "
May 26 11:48:23.086758 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.086534    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-isvc-logger-kube-rbac-proxy-sar-config\") pod \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") "
May 26 11:48:23.086758 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.086560    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kserve-provision-location\") pod \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\" (UID: \"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266\") "
May 26 11:48:23.086944 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.086915    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" (UID: "9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:48:23.087001 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.086954    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" (UID: "9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:48:23.088784 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.088755    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" (UID: "9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:48:23.088899 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.088820    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kube-api-access-mtg56" (OuterVolumeSpecName: "kube-api-access-mtg56") pod "9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" (UID: "9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266"). InnerVolumeSpecName "kube-api-access-mtg56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:48:23.187145 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.187071    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:48:23.187145 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.187100    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtg56\" (UniqueName: \"kubernetes.io/projected/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kube-api-access-mtg56\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:48:23.187145 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.187111    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:48:23.187145 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.187120    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:48:23.620131 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.620092    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p" event={"ID":"9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266","Type":"ContainerDied","Data":"39f6a818b9163bae13b9b0a4ae6f002973df970a44a68d1be7e4a3244f82acef"}
May 26 11:48:23.620131 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.620138    2563 scope.go:117] "RemoveContainer" containerID="dcdc05ae397ee3b32bc3f51d0095184064ac5523b7e236faba14bb324360e0ed"
May 26 11:48:23.620610 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.620170    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"
May 26 11:48:23.630180 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.628870    2563 scope.go:117] "RemoveContainer" containerID="9fb251df325ede3d7e561f42d2f7bfce3434ca55de33bc03b5fa319bdd6df7b4"
May 26 11:48:23.638025 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.638007    2563 scope.go:117] "RemoveContainer" containerID="08b44ac84a1c8023ad34b6646e8cb862d82068380ffb7800abe8a398eec7c6e7"
May 26 11:48:23.644844 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.644642    2563 scope.go:117] "RemoveContainer" containerID="81e44783406d57fb58b62455f42de0e00409d4707a6445ca74acc5a9108b257d"
May 26 11:48:23.646143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.646123    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"]
May 26 11:48:23.650322 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:23.650300    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-g944p"]
May 26 11:48:24.242619 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:24.242580    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" path="/var/lib/kubelet/pods/9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266/volumes"
May 26 11:48:30.571234 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:30.571184    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:48:40.571857 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:40.571817    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:48:50.571934 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:48:50.571895    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:49:00.571748 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:00.571707    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:49:10.571225 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:10.571184    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:49:11.238374 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:11.238325    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:49:21.239459 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:21.239426    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:49:22.459780 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.459740    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"]
May 26 11:49:22.460248 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.460156    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container" containerID="cri-o://d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d" gracePeriod=30
May 26 11:49:22.460315 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.460248    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kube-rbac-proxy" containerID="cri-o://720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148" gracePeriod=30
May 26 11:49:22.594850 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.594815    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"]
May 26 11:49:22.595129 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595113    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy"
May 26 11:49:22.595205 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595132    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy"
May 26 11:49:22.595205 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595150    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="storage-initializer"
May 26 11:49:22.595205 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595159    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="storage-initializer"
May 26 11:49:22.595205 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595169    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerName="kube-rbac-proxy"
May 26 11:49:22.595205 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595178    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerName="kube-rbac-proxy"
May 26 11:49:22.595205 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595191    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerName="kserve-container"
May 26 11:49:22.595205 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595199    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerName="kserve-container"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595209    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595218    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595229    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595238    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595306    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="agent"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595318    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerName="kserve-container"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595329    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kserve-container"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595339    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="44859f60-6eed-49bd-9550-ca2e825cd47b" containerName="kube-rbac-proxy"
May 26 11:49:22.595541 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.595351    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fdfc4d2-ce64-4960-9cf9-bfbd99fc7266" containerName="kube-rbac-proxy"
May 26 11:49:22.598423 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.598404    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.600413 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.600385    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
May 26 11:49:22.600781 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.600763    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
May 26 11:49:22.608313 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.608291    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"]
May 26 11:49:22.615424 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.615400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d47b73e1-913e-4b47-93e3-4579b121b9f1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.615508 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.615452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.615551 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.615500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgq4g\" (UniqueName: \"kubernetes.io/projected/d47b73e1-913e-4b47-93e3-4579b121b9f1-kube-api-access-qgq4g\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.615586 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.615547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47b73e1-913e-4b47-93e3-4579b121b9f1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.716059 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.715981    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qgq4g\" (UniqueName: \"kubernetes.io/projected/d47b73e1-913e-4b47-93e3-4579b121b9f1-kube-api-access-qgq4g\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.716059 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.716026    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47b73e1-913e-4b47-93e3-4579b121b9f1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.716059 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.716059    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d47b73e1-913e-4b47-93e3-4579b121b9f1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.716343 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.716185    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.716343 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:49:22.716296    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 26 11:49:22.716433 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:49:22.716362    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls podName:d47b73e1-913e-4b47-93e3-4579b121b9f1 nodeName:}" failed. No retries permitted until 2026-05-26 11:49:23.216340427 +0000 UTC m=+1325.579290094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls") pod "isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" (UID: "d47b73e1-913e-4b47-93e3-4579b121b9f1") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 26 11:49:22.716433 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.716412    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47b73e1-913e-4b47-93e3-4579b121b9f1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.716724 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.716705    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d47b73e1-913e-4b47-93e3-4579b121b9f1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.725705 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.725675    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgq4g\" (UniqueName: \"kubernetes.io/projected/d47b73e1-913e-4b47-93e3-4579b121b9f1-kube-api-access-qgq4g\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:22.775059 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.775032    2563 generic.go:358] "Generic (PLEG): container finished" podID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerID="720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148" exitCode=2
May 26 11:49:22.775188 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:22.775096    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" event={"ID":"e397c5f5-7263-4bc7-b38f-2ec61b4b9596","Type":"ContainerDied","Data":"720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148"}
May 26 11:49:23.220576 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:23.220541    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:23.223092 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:23.223070    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:23.509097 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:23.509009    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:23.635279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:23.635249    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"]
May 26 11:49:23.638248 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:49:23.638215    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47b73e1_913e_4b47_93e3_4579b121b9f1.slice/crio-ab7cd794b7077fbb18466259e68a01df865678b647a95fd124f2cab5873d99af WatchSource:0}: Error finding container ab7cd794b7077fbb18466259e68a01df865678b647a95fd124f2cab5873d99af: Status 404 returned error can't find the container with id ab7cd794b7077fbb18466259e68a01df865678b647a95fd124f2cab5873d99af
May 26 11:49:23.782814 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:23.780204    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" event={"ID":"d47b73e1-913e-4b47-93e3-4579b121b9f1","Type":"ContainerStarted","Data":"9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4"}
May 26 11:49:23.782814 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:23.780258    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" event={"ID":"d47b73e1-913e-4b47-93e3-4579b121b9f1","Type":"ContainerStarted","Data":"ab7cd794b7077fbb18466259e68a01df865678b647a95fd124f2cab5873d99af"}
May 26 11:49:25.566787 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:25.566751    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
May 26 11:49:27.604397 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.604374    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:49:27.654096 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.654064    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") "
May 26 11:49:27.654297 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.654119    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-proxy-tls\") pod \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") "
May 26 11:49:27.654297 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.654142    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph9j9\" (UniqueName: \"kubernetes.io/projected/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kube-api-access-ph9j9\") pod \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") "
May 26 11:49:27.654297 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.654161    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kserve-provision-location\") pod \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\" (UID: \"e397c5f5-7263-4bc7-b38f-2ec61b4b9596\") "
May 26 11:49:27.654521 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.654486    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "e397c5f5-7263-4bc7-b38f-2ec61b4b9596" (UID: "e397c5f5-7263-4bc7-b38f-2ec61b4b9596"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:49:27.654628 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.654531    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e397c5f5-7263-4bc7-b38f-2ec61b4b9596" (UID: "e397c5f5-7263-4bc7-b38f-2ec61b4b9596"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:49:27.656274 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.656244    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e397c5f5-7263-4bc7-b38f-2ec61b4b9596" (UID: "e397c5f5-7263-4bc7-b38f-2ec61b4b9596"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:49:27.656361 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.656306    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kube-api-access-ph9j9" (OuterVolumeSpecName: "kube-api-access-ph9j9") pod "e397c5f5-7263-4bc7-b38f-2ec61b4b9596" (UID: "e397c5f5-7263-4bc7-b38f-2ec61b4b9596"). InnerVolumeSpecName "kube-api-access-ph9j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:49:27.755429 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.755393    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:49:27.755429 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.755424    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9j9\" (UniqueName: \"kubernetes.io/projected/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kube-api-access-ph9j9\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:49:27.755626 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.755438    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:49:27.755626 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.755455    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e397c5f5-7263-4bc7-b38f-2ec61b4b9596-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:49:27.792693 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.792664    2563 generic.go:358] "Generic (PLEG): container finished" podID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerID="9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4" exitCode=0
May 26 11:49:27.792865 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.792744    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" event={"ID":"d47b73e1-913e-4b47-93e3-4579b121b9f1","Type":"ContainerDied","Data":"9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4"}
May 26 11:49:27.794332 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.794311    2563 generic.go:358] "Generic (PLEG): container finished" podID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerID="d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d" exitCode=0
May 26 11:49:27.794407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.794362    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" event={"ID":"e397c5f5-7263-4bc7-b38f-2ec61b4b9596","Type":"ContainerDied","Data":"d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d"}
May 26 11:49:27.794407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.794380    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72" event={"ID":"e397c5f5-7263-4bc7-b38f-2ec61b4b9596","Type":"ContainerDied","Data":"51a8e855d23886ea75cfade8d5214ae7aa132227ff43cf66367c95afc156cdda"}
May 26 11:49:27.794407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.794383    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"
May 26 11:49:27.794407 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.794396    2563 scope.go:117] "RemoveContainer" containerID="720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148"
May 26 11:49:27.802230 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.802172    2563 scope.go:117] "RemoveContainer" containerID="d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d"
May 26 11:49:27.809229 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.809209    2563 scope.go:117] "RemoveContainer" containerID="d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046"
May 26 11:49:27.816775 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.816755    2563 scope.go:117] "RemoveContainer" containerID="720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148"
May 26 11:49:27.817081 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:49:27.817058    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148\": container with ID starting with 720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148 not found: ID does not exist" containerID="720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148"
May 26 11:49:27.817143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.817097    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148"} err="failed to get container status \"720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148\": rpc error: code = NotFound desc = could not find container \"720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148\": container with ID starting with 720b7e901ea514fd21611b2cb85be7399bd00a8e93abd83b2d6597e1b1dbf148 not found: ID does not exist"
May 26 11:49:27.817143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.817121    2563 scope.go:117] "RemoveContainer" containerID="d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d"
May 26 11:49:27.817326 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:49:27.817307    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d\": container with ID starting with d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d not found: ID does not exist" containerID="d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d"
May 26 11:49:27.817382 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.817338    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d"} err="failed to get container status \"d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d\": rpc error: code = NotFound desc = could not find container \"d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d\": container with ID starting with d4e1e88ac3529a98256cd6dd03198fe7728bad771bf2f2aeee3e5da2fef42b9d not found: ID does not exist"
May 26 11:49:27.817382 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.817362    2563 scope.go:117] "RemoveContainer" containerID="d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046"
May 26 11:49:27.817585 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:49:27.817568    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046\": container with ID starting with d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046 not found: ID does not exist" containerID="d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046"
May 26 11:49:27.817585 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.817591    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046"} err="failed to get container status \"d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046\": rpc error: code = NotFound desc = could not find container \"d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046\": container with ID starting with d50e874641275f298977fa78389caef943bfa2ebf3619f566032b074e468f046 not found: ID does not exist"
May 26 11:49:27.823872 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.823851    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"]
May 26 11:49:27.835235 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:27.835210    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-mwn72"]
May 26 11:49:28.241688 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:28.241609    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" path="/var/lib/kubelet/pods/e397c5f5-7263-4bc7-b38f-2ec61b4b9596/volumes"
May 26 11:49:28.798593 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:28.798553    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" event={"ID":"d47b73e1-913e-4b47-93e3-4579b121b9f1","Type":"ContainerStarted","Data":"8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565"}
May 26 11:49:28.799095 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:28.798600    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" event={"ID":"d47b73e1-913e-4b47-93e3-4579b121b9f1","Type":"ContainerStarted","Data":"3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db"}
May 26 11:49:28.799095 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:28.798996    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:28.799095 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:28.799034    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:28.800359 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:28.800328    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:49:28.818150 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:28.818098    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podStartSLOduration=6.81808471 podStartE2EDuration="6.81808471s" podCreationTimestamp="2026-05-26 11:49:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:49:28.81804408 +0000 UTC m=+1331.180993751" watchObservedRunningTime="2026-05-26 11:49:28.81808471 +0000 UTC m=+1331.181034374"
May 26 11:49:29.802486 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:29.802438    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:49:34.806911 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:34.806882    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:49:34.807390 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:34.807364    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:49:44.807533 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:44.807443    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:49:54.807853 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:49:54.807788    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:50:04.808297 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:04.808253    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:50:14.807840 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:14.807779    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:50:24.808115 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:24.808071    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:50:34.807813 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:34.807748    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:50:44.807943 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:44.807914    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:50:53.288666 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.288632    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"]
May 26 11:50:53.289065 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.288964    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" containerID="cri-o://3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db" gracePeriod=30
May 26 11:50:53.289139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.289028    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kube-rbac-proxy" containerID="cri-o://8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565" gracePeriod=30
May 26 11:50:53.423527 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423495    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"]
May 26 11:50:53.423756 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423744    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container"
May 26 11:50:53.423821 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423758    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container"
May 26 11:50:53.423821 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423768    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="storage-initializer"
May 26 11:50:53.423821 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423774    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="storage-initializer"
May 26 11:50:53.423821 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423788    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kube-rbac-proxy"
May 26 11:50:53.423821 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423808    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kube-rbac-proxy"
May 26 11:50:53.423971 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423852    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kube-rbac-proxy"
May 26 11:50:53.423971 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.423860    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e397c5f5-7263-4bc7-b38f-2ec61b4b9596" containerName="kserve-container"
May 26 11:50:53.426616 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.426600    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.432423 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.432281    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
May 26 11:50:53.432664 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.432620    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
May 26 11:50:53.439372 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.439352    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"]
May 26 11:50:53.574833 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.574787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2697dd3-d115-42a5-a374-0350e3cb629f-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.574983 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.574841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2697dd3-d115-42a5-a374-0350e3cb629f-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.574983 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.574899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xcnn\" (UniqueName: \"kubernetes.io/projected/e2697dd3-d115-42a5-a374-0350e3cb629f-kube-api-access-2xcnn\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.574983 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.574976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2697dd3-d115-42a5-a374-0350e3cb629f-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.676283 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.676244    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2697dd3-d115-42a5-a374-0350e3cb629f-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.676283 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.676281    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2697dd3-d115-42a5-a374-0350e3cb629f-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.676490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.676305    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xcnn\" (UniqueName: \"kubernetes.io/projected/e2697dd3-d115-42a5-a374-0350e3cb629f-kube-api-access-2xcnn\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.676490 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.676379    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2697dd3-d115-42a5-a374-0350e3cb629f-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.676843 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.676821    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2697dd3-d115-42a5-a374-0350e3cb629f-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.677032 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.677016    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2697dd3-d115-42a5-a374-0350e3cb629f-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.678958 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.678938    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2697dd3-d115-42a5-a374-0350e3cb629f-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.684423 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.684397    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xcnn\" (UniqueName: \"kubernetes.io/projected/e2697dd3-d115-42a5-a374-0350e3cb629f-kube-api-access-2xcnn\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.736421 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.736385    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:50:53.866067 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:53.865978    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"]
May 26 11:50:53.868871 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:50:53.868841    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2697dd3_d115_42a5_a374_0350e3cb629f.slice/crio-50e2e6c363cd5acd12b9bd9c8247d8af2596cdc61f00c196bd06ffb22620071d WatchSource:0}: Error finding container 50e2e6c363cd5acd12b9bd9c8247d8af2596cdc61f00c196bd06ffb22620071d: Status 404 returned error can't find the container with id 50e2e6c363cd5acd12b9bd9c8247d8af2596cdc61f00c196bd06ffb22620071d
May 26 11:50:54.025239 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:54.025206    2563 generic.go:358] "Generic (PLEG): container finished" podID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerID="8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565" exitCode=2
May 26 11:50:54.025431 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:54.025284    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" event={"ID":"d47b73e1-913e-4b47-93e3-4579b121b9f1","Type":"ContainerDied","Data":"8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565"}
May 26 11:50:54.026716 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:54.026683    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" event={"ID":"e2697dd3-d115-42a5-a374-0350e3cb629f","Type":"ContainerStarted","Data":"074fbb5299d6b56566dccb0e77d1ebfe9ab6d49ee8bbb0a4ac660d257e21d6c4"}
May 26 11:50:54.026716 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:54.026716    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" event={"ID":"e2697dd3-d115-42a5-a374-0350e3cb629f","Type":"ContainerStarted","Data":"50e2e6c363cd5acd12b9bd9c8247d8af2596cdc61f00c196bd06ffb22620071d"}
May 26 11:50:54.803221 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:54.803179    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 26 11:50:54.807512 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:54.807486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:50:58.039234 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.039146    2563 generic.go:358] "Generic (PLEG): container finished" podID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerID="074fbb5299d6b56566dccb0e77d1ebfe9ab6d49ee8bbb0a4ac660d257e21d6c4" exitCode=0
May 26 11:50:58.039712 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.039225    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" event={"ID":"e2697dd3-d115-42a5-a374-0350e3cb629f","Type":"ContainerDied","Data":"074fbb5299d6b56566dccb0e77d1ebfe9ab6d49ee8bbb0a4ac660d257e21d6c4"}
May 26 11:50:58.681991 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.681707    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:50:58.816279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.815541    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgq4g\" (UniqueName: \"kubernetes.io/projected/d47b73e1-913e-4b47-93e3-4579b121b9f1-kube-api-access-qgq4g\") pod \"d47b73e1-913e-4b47-93e3-4579b121b9f1\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") "
May 26 11:50:58.816279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.815635    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d47b73e1-913e-4b47-93e3-4579b121b9f1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"d47b73e1-913e-4b47-93e3-4579b121b9f1\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") "
May 26 11:50:58.816279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.815698    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls\") pod \"d47b73e1-913e-4b47-93e3-4579b121b9f1\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") "
May 26 11:50:58.816279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.815725    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47b73e1-913e-4b47-93e3-4579b121b9f1-kserve-provision-location\") pod \"d47b73e1-913e-4b47-93e3-4579b121b9f1\" (UID: \"d47b73e1-913e-4b47-93e3-4579b121b9f1\") "
May 26 11:50:58.816837 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.816655    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47b73e1-913e-4b47-93e3-4579b121b9f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d47b73e1-913e-4b47-93e3-4579b121b9f1" (UID: "d47b73e1-913e-4b47-93e3-4579b121b9f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:50:58.816837 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.816781    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47b73e1-913e-4b47-93e3-4579b121b9f1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "d47b73e1-913e-4b47-93e3-4579b121b9f1" (UID: "d47b73e1-913e-4b47-93e3-4579b121b9f1"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:50:58.821599 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.821568    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d47b73e1-913e-4b47-93e3-4579b121b9f1" (UID: "d47b73e1-913e-4b47-93e3-4579b121b9f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:50:58.823896 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.823868    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47b73e1-913e-4b47-93e3-4579b121b9f1-kube-api-access-qgq4g" (OuterVolumeSpecName: "kube-api-access-qgq4g") pod "d47b73e1-913e-4b47-93e3-4579b121b9f1" (UID: "d47b73e1-913e-4b47-93e3-4579b121b9f1"). InnerVolumeSpecName "kube-api-access-qgq4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:50:58.917231 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.917019    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d47b73e1-913e-4b47-93e3-4579b121b9f1-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:50:58.917231 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.917055    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47b73e1-913e-4b47-93e3-4579b121b9f1-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:50:58.917231 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.917076    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgq4g\" (UniqueName: \"kubernetes.io/projected/d47b73e1-913e-4b47-93e3-4579b121b9f1-kube-api-access-qgq4g\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:50:58.917231 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:58.917092    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d47b73e1-913e-4b47-93e3-4579b121b9f1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:50:59.047698 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.046058    2563 generic.go:358] "Generic (PLEG): container finished" podID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerID="3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db" exitCode=0
May 26 11:50:59.047698 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.046130    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" event={"ID":"d47b73e1-913e-4b47-93e3-4579b121b9f1","Type":"ContainerDied","Data":"3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db"}
May 26 11:50:59.047698 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.046160    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k" event={"ID":"d47b73e1-913e-4b47-93e3-4579b121b9f1","Type":"ContainerDied","Data":"ab7cd794b7077fbb18466259e68a01df865678b647a95fd124f2cab5873d99af"}
May 26 11:50:59.047698 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.046179    2563 scope.go:117] "RemoveContainer" containerID="8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565"
May 26 11:50:59.047698 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.046355    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"
May 26 11:50:59.066714 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.066628    2563 scope.go:117] "RemoveContainer" containerID="3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db"
May 26 11:50:59.078042 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.078013    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"]
May 26 11:50:59.080714 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.080696    2563 scope.go:117] "RemoveContainer" containerID="9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4"
May 26 11:50:59.083959 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.083937    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jbr8k"]
May 26 11:50:59.101785 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.101759    2563 scope.go:117] "RemoveContainer" containerID="8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565"
May 26 11:50:59.102335 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:50:59.102164    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565\": container with ID starting with 8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565 not found: ID does not exist" containerID="8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565"
May 26 11:50:59.102335 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.102200    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565"} err="failed to get container status \"8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565\": rpc error: code = NotFound desc = could not find container \"8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565\": container with ID starting with 8a2b4170280f3efc1673283ec325f62dd0c796cf8d1796ef0c6ef951d7aaf565 not found: ID does not exist"
May 26 11:50:59.102335 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.102226    2563 scope.go:117] "RemoveContainer" containerID="3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db"
May 26 11:50:59.103473 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:50:59.103331    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db\": container with ID starting with 3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db not found: ID does not exist" containerID="3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db"
May 26 11:50:59.103473 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.103362    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db"} err="failed to get container status \"3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db\": rpc error: code = NotFound desc = could not find container \"3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db\": container with ID starting with 3f7c91c087961424a8079d2bb6a2e7cdf75553d774d93be61b81888605d716db not found: ID does not exist"
May 26 11:50:59.103473 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.103382    2563 scope.go:117] "RemoveContainer" containerID="9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4"
May 26 11:50:59.103903 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:50:59.103817    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4\": container with ID starting with 9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4 not found: ID does not exist" containerID="9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4"
May 26 11:50:59.103903 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:50:59.103845    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4"} err="failed to get container status \"9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4\": rpc error: code = NotFound desc = could not find container \"9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4\": container with ID starting with 9ee2a38af10df286f2bf0018a7cc285cde7d830eb339efc0d8b237640c489ff4 not found: ID does not exist"
May 26 11:51:00.243580 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:51:00.243542    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" path="/var/lib/kubelet/pods/d47b73e1-913e-4b47-93e3-4579b121b9f1/volumes"
May 26 11:53:01.344965 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:01.344933    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:53:01.345515 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:01.344982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:53:02.025463 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:02.025439    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:53:02.403688 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:02.403649    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" event={"ID":"e2697dd3-d115-42a5-a374-0350e3cb629f","Type":"ContainerStarted","Data":"7bcdc58c55d68cee10f73ea5500bc4340cc6d5f2168657df7e4d98d065c2425d"}
May 26 11:53:02.403688 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:02.403692    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" event={"ID":"e2697dd3-d115-42a5-a374-0350e3cb629f","Type":"ContainerStarted","Data":"1564e452d146065b97d3f3402b61a803273859b1a48c7170e7d4dfbdb230b6f6"}
May 26 11:53:02.404097 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:02.403834    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:53:02.436320 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:02.436261    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" podStartSLOduration=5.577918204 podStartE2EDuration="2m9.436243222s" podCreationTimestamp="2026-05-26 11:50:53 +0000 UTC" firstStartedPulling="2026-05-26 11:50:58.040446693 +0000 UTC m=+1420.403396340" lastFinishedPulling="2026-05-26 11:53:01.898771708 +0000 UTC m=+1544.261721358" observedRunningTime="2026-05-26 11:53:02.434923149 +0000 UTC m=+1544.797872817" watchObservedRunningTime="2026-05-26 11:53:02.436243222 +0000 UTC m=+1544.799192892"
May 26 11:53:03.406739 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:03.406704    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:53:09.414204 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:09.414175    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:53:39.418184 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:39.418152    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:53:43.599302 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.599266    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"]
May 26 11:53:43.599781 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.599565    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kserve-container" containerID="cri-o://1564e452d146065b97d3f3402b61a803273859b1a48c7170e7d4dfbdb230b6f6" gracePeriod=30
May 26 11:53:43.599781 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.599604    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kube-rbac-proxy" containerID="cri-o://7bcdc58c55d68cee10f73ea5500bc4340cc6d5f2168657df7e4d98d065c2425d" gracePeriod=30
May 26 11:53:43.707701 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.707667    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"]
May 26 11:53:43.708070 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.708050    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kube-rbac-proxy"
May 26 11:53:43.708139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.708075    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kube-rbac-proxy"
May 26 11:53:43.708139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.708089    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container"
May 26 11:53:43.708139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.708098    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container"
May 26 11:53:43.708139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.708131    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="storage-initializer"
May 26 11:53:43.708139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.708139    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="storage-initializer"
May 26 11:53:43.708306 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.708203    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kserve-container"
May 26 11:53:43.708306 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.708216    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d47b73e1-913e-4b47-93e3-4579b121b9f1" containerName="kube-rbac-proxy"
May 26 11:53:43.710397 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.710373    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.712429 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.712404    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
May 26 11:53:43.713062 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.712766    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
May 26 11:53:43.720483 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.720454    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"]
May 26 11:53:43.820025 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.819985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d268c6e7-03be-49b6-8478-34d39e590ffb-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.820209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.820041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d268c6e7-03be-49b6-8478-34d39e590ffb-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.820209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.820064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlkzd\" (UniqueName: \"kubernetes.io/projected/d268c6e7-03be-49b6-8478-34d39e590ffb-kube-api-access-rlkzd\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.820209 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.820174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d268c6e7-03be-49b6-8478-34d39e590ffb-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.921064 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.920980    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d268c6e7-03be-49b6-8478-34d39e590ffb-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.921064 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.921027    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rlkzd\" (UniqueName: \"kubernetes.io/projected/d268c6e7-03be-49b6-8478-34d39e590ffb-kube-api-access-rlkzd\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.921279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.921069    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d268c6e7-03be-49b6-8478-34d39e590ffb-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.921279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.921256    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d268c6e7-03be-49b6-8478-34d39e590ffb-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.921625 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.921595    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d268c6e7-03be-49b6-8478-34d39e590ffb-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.921752 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.921730    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d268c6e7-03be-49b6-8478-34d39e590ffb-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.923665 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.923647    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d268c6e7-03be-49b6-8478-34d39e590ffb-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:43.931089 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:43.931062    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlkzd\" (UniqueName: \"kubernetes.io/projected/d268c6e7-03be-49b6-8478-34d39e590ffb-kube-api-access-rlkzd\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:44.022117 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.022089    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:44.148764 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.148721    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"]
May 26 11:53:44.151852 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:53:44.151824    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd268c6e7_03be_49b6_8478_34d39e590ffb.slice/crio-6d9b14ae5510520e9a67759323c5cbef2edb1323aa67d994faa98b27a0e8ccea WatchSource:0}: Error finding container 6d9b14ae5510520e9a67759323c5cbef2edb1323aa67d994faa98b27a0e8ccea: Status 404 returned error can't find the container with id 6d9b14ae5510520e9a67759323c5cbef2edb1323aa67d994faa98b27a0e8ccea
May 26 11:53:44.409945 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.409905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 26 11:53:44.515427 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.515398    2563 generic.go:358] "Generic (PLEG): container finished" podID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerID="7bcdc58c55d68cee10f73ea5500bc4340cc6d5f2168657df7e4d98d065c2425d" exitCode=2
May 26 11:53:44.515427 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.515422    2563 generic.go:358] "Generic (PLEG): container finished" podID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerID="1564e452d146065b97d3f3402b61a803273859b1a48c7170e7d4dfbdb230b6f6" exitCode=0
May 26 11:53:44.515647 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.515467    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" event={"ID":"e2697dd3-d115-42a5-a374-0350e3cb629f","Type":"ContainerDied","Data":"7bcdc58c55d68cee10f73ea5500bc4340cc6d5f2168657df7e4d98d065c2425d"}
May 26 11:53:44.515647 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.515570    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" event={"ID":"e2697dd3-d115-42a5-a374-0350e3cb629f","Type":"ContainerDied","Data":"1564e452d146065b97d3f3402b61a803273859b1a48c7170e7d4dfbdb230b6f6"}
May 26 11:53:44.517179 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.517159    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" event={"ID":"d268c6e7-03be-49b6-8478-34d39e590ffb","Type":"ContainerStarted","Data":"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057"}
May 26 11:53:44.517279 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.517184    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" event={"ID":"d268c6e7-03be-49b6-8478-34d39e590ffb","Type":"ContainerStarted","Data":"6d9b14ae5510520e9a67759323c5cbef2edb1323aa67d994faa98b27a0e8ccea"}
May 26 11:53:44.640563 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.640539    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:53:44.727510 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.727432    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2697dd3-d115-42a5-a374-0350e3cb629f-proxy-tls\") pod \"e2697dd3-d115-42a5-a374-0350e3cb629f\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") "
May 26 11:53:44.727510 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.727470    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2697dd3-d115-42a5-a374-0350e3cb629f-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"e2697dd3-d115-42a5-a374-0350e3cb629f\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") "
May 26 11:53:44.727510 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.727506    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xcnn\" (UniqueName: \"kubernetes.io/projected/e2697dd3-d115-42a5-a374-0350e3cb629f-kube-api-access-2xcnn\") pod \"e2697dd3-d115-42a5-a374-0350e3cb629f\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") "
May 26 11:53:44.727707 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.727526    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2697dd3-d115-42a5-a374-0350e3cb629f-kserve-provision-location\") pod \"e2697dd3-d115-42a5-a374-0350e3cb629f\" (UID: \"e2697dd3-d115-42a5-a374-0350e3cb629f\") "
May 26 11:53:44.727871 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.727846    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2697dd3-d115-42a5-a374-0350e3cb629f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2697dd3-d115-42a5-a374-0350e3cb629f" (UID: "e2697dd3-d115-42a5-a374-0350e3cb629f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:53:44.727988 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.727891    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2697dd3-d115-42a5-a374-0350e3cb629f-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "e2697dd3-d115-42a5-a374-0350e3cb629f" (UID: "e2697dd3-d115-42a5-a374-0350e3cb629f"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:53:44.729585 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.729561    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2697dd3-d115-42a5-a374-0350e3cb629f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2697dd3-d115-42a5-a374-0350e3cb629f" (UID: "e2697dd3-d115-42a5-a374-0350e3cb629f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:53:44.729675 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.729657    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2697dd3-d115-42a5-a374-0350e3cb629f-kube-api-access-2xcnn" (OuterVolumeSpecName: "kube-api-access-2xcnn") pod "e2697dd3-d115-42a5-a374-0350e3cb629f" (UID: "e2697dd3-d115-42a5-a374-0350e3cb629f"). InnerVolumeSpecName "kube-api-access-2xcnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:53:44.828123 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.828090    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xcnn\" (UniqueName: \"kubernetes.io/projected/e2697dd3-d115-42a5-a374-0350e3cb629f-kube-api-access-2xcnn\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:53:44.828123 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.828120    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2697dd3-d115-42a5-a374-0350e3cb629f-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:53:44.828123 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.828131    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2697dd3-d115-42a5-a374-0350e3cb629f-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:53:44.828305 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:44.828140    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2697dd3-d115-42a5-a374-0350e3cb629f-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:53:45.521143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:45.521115    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"
May 26 11:53:45.521143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:45.521130    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b" event={"ID":"e2697dd3-d115-42a5-a374-0350e3cb629f","Type":"ContainerDied","Data":"50e2e6c363cd5acd12b9bd9c8247d8af2596cdc61f00c196bd06ffb22620071d"}
May 26 11:53:45.521361 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:45.521169    2563 scope.go:117] "RemoveContainer" containerID="7bcdc58c55d68cee10f73ea5500bc4340cc6d5f2168657df7e4d98d065c2425d"
May 26 11:53:45.529188 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:45.529064    2563 scope.go:117] "RemoveContainer" containerID="1564e452d146065b97d3f3402b61a803273859b1a48c7170e7d4dfbdb230b6f6"
May 26 11:53:45.535833 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:45.535816    2563 scope.go:117] "RemoveContainer" containerID="074fbb5299d6b56566dccb0e77d1ebfe9ab6d49ee8bbb0a4ac660d257e21d6c4"
May 26 11:53:45.543395 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:45.543375    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"]
May 26 11:53:45.548403 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:45.548382    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-n6b8b"]
May 26 11:53:46.241447 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:46.241415    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" path="/var/lib/kubelet/pods/e2697dd3-d115-42a5-a374-0350e3cb629f/volumes"
May 26 11:53:48.531017 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:48.530988    2563 generic.go:358] "Generic (PLEG): container finished" podID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerID="c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057" exitCode=0
May 26 11:53:48.531393 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:48.531069    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" event={"ID":"d268c6e7-03be-49b6-8478-34d39e590ffb","Type":"ContainerDied","Data":"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057"}
May 26 11:53:49.536130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:49.536088    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" event={"ID":"d268c6e7-03be-49b6-8478-34d39e590ffb","Type":"ContainerStarted","Data":"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219"}
May 26 11:53:49.536452 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:49.536138    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" event={"ID":"d268c6e7-03be-49b6-8478-34d39e590ffb","Type":"ContainerStarted","Data":"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6"}
May 26 11:53:49.536452 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:49.536349    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:49.555128 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:49.555086    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" podStartSLOduration=6.555068333 podStartE2EDuration="6.555068333s" podCreationTimestamp="2026-05-26 11:53:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:53:49.554025274 +0000 UTC m=+1591.916974943" watchObservedRunningTime="2026-05-26 11:53:49.555068333 +0000 UTC m=+1591.918018004"
May 26 11:53:50.538542 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:50.538506    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:50.540021 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:50.539994    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:53:51.541700 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:51.541657    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:53:56.545997 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:56.545966    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:53:56.546905 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:53:56.546886    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:54:03.736390 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.736358    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"]
May 26 11:54:03.736816 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.736703    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kserve-container" containerID="cri-o://a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6" gracePeriod=30
May 26 11:54:03.737063 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.736995    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kube-rbac-proxy" containerID="cri-o://cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219" gracePeriod=30
May 26 11:54:03.813947 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.813910    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"]
May 26 11:54:03.814247 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.814230    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kube-rbac-proxy"
May 26 11:54:03.814321 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.814250    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kube-rbac-proxy"
May 26 11:54:03.814321 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.814275    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="storage-initializer"
May 26 11:54:03.814321 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.814284    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="storage-initializer"
May 26 11:54:03.814321 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.814301    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kserve-container"
May 26 11:54:03.814321 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.814310    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kserve-container"
May 26 11:54:03.814562 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.814376    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kserve-container"
May 26 11:54:03.814562 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.814389    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2697dd3-d115-42a5-a374-0350e3cb629f" containerName="kube-rbac-proxy"
May 26 11:54:03.816295 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.816275    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:03.818859 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.818843    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
May 26 11:54:03.819079 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.819055    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
May 26 11:54:03.830445 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.830422    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"]
May 26 11:54:03.971999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.971961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdm7m\" (UniqueName: \"kubernetes.io/projected/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kube-api-access-jdm7m\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:03.971999 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.972002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d6b1d-a6cd-4353-864b-11b322f5be23-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:03.972246 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.972043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:03.972246 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:03.972115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d6b1d-a6cd-4353-864b-11b322f5be23-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.072652 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.072614    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.072881 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.072682    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d6b1d-a6cd-4353-864b-11b322f5be23-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.072881 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.072709    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jdm7m\" (UniqueName: \"kubernetes.io/projected/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kube-api-access-jdm7m\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.072881 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.072727    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d6b1d-a6cd-4353-864b-11b322f5be23-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.073130 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.073107    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.073421 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.073398    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d6b1d-a6cd-4353-864b-11b322f5be23-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.075455 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.075434    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d6b1d-a6cd-4353-864b-11b322f5be23-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.081499 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.081475    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdm7m\" (UniqueName: \"kubernetes.io/projected/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kube-api-access-jdm7m\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.126371 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.126315    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:04.258269 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.258215    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"]
May 26 11:54:04.262246 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:54:04.262219    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3d6b1d_a6cd_4353_864b_11b322f5be23.slice/crio-31b8d462e8fdde644d78082a2224a7129cbe787ed27c7ca1a20d7a3c77d98a5e WatchSource:0}: Error finding container 31b8d462e8fdde644d78082a2224a7129cbe787ed27c7ca1a20d7a3c77d98a5e: Status 404 returned error can't find the container with id 31b8d462e8fdde644d78082a2224a7129cbe787ed27c7ca1a20d7a3c77d98a5e
May 26 11:54:04.368124 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.368099    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:54:04.475111 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.475076    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d268c6e7-03be-49b6-8478-34d39e590ffb-kserve-provision-location\") pod \"d268c6e7-03be-49b6-8478-34d39e590ffb\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") "
May 26 11:54:04.475335 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.475150    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d268c6e7-03be-49b6-8478-34d39e590ffb-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"d268c6e7-03be-49b6-8478-34d39e590ffb\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") "
May 26 11:54:04.475335 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.475172    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d268c6e7-03be-49b6-8478-34d39e590ffb-proxy-tls\") pod \"d268c6e7-03be-49b6-8478-34d39e590ffb\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") "
May 26 11:54:04.475335 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.475197    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlkzd\" (UniqueName: \"kubernetes.io/projected/d268c6e7-03be-49b6-8478-34d39e590ffb-kube-api-access-rlkzd\") pod \"d268c6e7-03be-49b6-8478-34d39e590ffb\" (UID: \"d268c6e7-03be-49b6-8478-34d39e590ffb\") "
May 26 11:54:04.475551 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.475519    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d268c6e7-03be-49b6-8478-34d39e590ffb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d268c6e7-03be-49b6-8478-34d39e590ffb" (UID: "d268c6e7-03be-49b6-8478-34d39e590ffb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:54:04.475640 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.475616    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d268c6e7-03be-49b6-8478-34d39e590ffb-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "d268c6e7-03be-49b6-8478-34d39e590ffb" (UID: "d268c6e7-03be-49b6-8478-34d39e590ffb"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:54:04.477445 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.477415    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d268c6e7-03be-49b6-8478-34d39e590ffb-kube-api-access-rlkzd" (OuterVolumeSpecName: "kube-api-access-rlkzd") pod "d268c6e7-03be-49b6-8478-34d39e590ffb" (UID: "d268c6e7-03be-49b6-8478-34d39e590ffb"). InnerVolumeSpecName "kube-api-access-rlkzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:54:04.477445 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.477433    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d268c6e7-03be-49b6-8478-34d39e590ffb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d268c6e7-03be-49b6-8478-34d39e590ffb" (UID: "d268c6e7-03be-49b6-8478-34d39e590ffb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:54:04.575773 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.575735    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d268c6e7-03be-49b6-8478-34d39e590ffb-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:54:04.575773 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.575765    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d268c6e7-03be-49b6-8478-34d39e590ffb-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:54:04.575956 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.575781    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlkzd\" (UniqueName: \"kubernetes.io/projected/d268c6e7-03be-49b6-8478-34d39e590ffb-kube-api-access-rlkzd\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:54:04.575956 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.575813    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d268c6e7-03be-49b6-8478-34d39e590ffb-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:54:04.579577 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.579539    2563 generic.go:358] "Generic (PLEG): container finished" podID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerID="cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219" exitCode=2
May 26 11:54:04.579577 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.579568    2563 generic.go:358] "Generic (PLEG): container finished" podID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerID="a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6" exitCode=0
May 26 11:54:04.579776 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.579615    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" event={"ID":"d268c6e7-03be-49b6-8478-34d39e590ffb","Type":"ContainerDied","Data":"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219"}
May 26 11:54:04.579776 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.579631    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"
May 26 11:54:04.579776 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.579652    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" event={"ID":"d268c6e7-03be-49b6-8478-34d39e590ffb","Type":"ContainerDied","Data":"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6"}
May 26 11:54:04.579776 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.579663    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg" event={"ID":"d268c6e7-03be-49b6-8478-34d39e590ffb","Type":"ContainerDied","Data":"6d9b14ae5510520e9a67759323c5cbef2edb1323aa67d994faa98b27a0e8ccea"}
May 26 11:54:04.579776 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.579677    2563 scope.go:117] "RemoveContainer" containerID="cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219"
May 26 11:54:04.581251 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.581150    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" event={"ID":"ce3d6b1d-a6cd-4353-864b-11b322f5be23","Type":"ContainerStarted","Data":"3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906"}
May 26 11:54:04.581251 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.581187    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" event={"ID":"ce3d6b1d-a6cd-4353-864b-11b322f5be23","Type":"ContainerStarted","Data":"31b8d462e8fdde644d78082a2224a7129cbe787ed27c7ca1a20d7a3c77d98a5e"}
May 26 11:54:04.587752 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.587703    2563 scope.go:117] "RemoveContainer" containerID="a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6"
May 26 11:54:04.594330 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.594311    2563 scope.go:117] "RemoveContainer" containerID="c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057"
May 26 11:54:04.600616 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.600599    2563 scope.go:117] "RemoveContainer" containerID="cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219"
May 26 11:54:04.600879 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:54:04.600856    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219\": container with ID starting with cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219 not found: ID does not exist" containerID="cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219"
May 26 11:54:04.600968 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.600891    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219"} err="failed to get container status \"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219\": rpc error: code = NotFound desc = could not find container \"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219\": container with ID starting with cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219 not found: ID does not exist"
May 26 11:54:04.600968 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.600912    2563 scope.go:117] "RemoveContainer" containerID="a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6"
May 26 11:54:04.601158 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:54:04.601139    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6\": container with ID starting with a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6 not found: ID does not exist" containerID="a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6"
May 26 11:54:04.601222 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.601167    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6"} err="failed to get container status \"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6\": rpc error: code = NotFound desc = could not find container \"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6\": container with ID starting with a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6 not found: ID does not exist"
May 26 11:54:04.601222 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.601189    2563 scope.go:117] "RemoveContainer" containerID="c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057"
May 26 11:54:04.601419 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:54:04.601404    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057\": container with ID starting with c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057 not found: ID does not exist" containerID="c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057"
May 26 11:54:04.601474 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.601426    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057"} err="failed to get container status \"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057\": rpc error: code = NotFound desc = could not find container \"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057\": container with ID starting with c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057 not found: ID does not exist"
May 26 11:54:04.601474 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.601444    2563 scope.go:117] "RemoveContainer" containerID="cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219"
May 26 11:54:04.601671 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.601653    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219"} err="failed to get container status \"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219\": rpc error: code = NotFound desc = could not find container \"cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219\": container with ID starting with cb03602164254647329313dcd555ae6399657ad47cd8477f6fb8064dafb93219 not found: ID does not exist"
May 26 11:54:04.601671 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.601670    2563 scope.go:117] "RemoveContainer" containerID="a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6"
May 26 11:54:04.601878 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.601860    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6"} err="failed to get container status \"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6\": rpc error: code = NotFound desc = could not find container \"a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6\": container with ID starting with a4892122942d865f5910cffba5f8cb16eb6036fec86b27a3742341bc862060f6 not found: ID does not exist"
May 26 11:54:04.601943 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.601880    2563 scope.go:117] "RemoveContainer" containerID="c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057"
May 26 11:54:04.602080 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.602057    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057"} err="failed to get container status \"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057\": rpc error: code = NotFound desc = could not find container \"c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057\": container with ID starting with c5c0cd2e50bd6e6139f88e7a7867d273afc60a508afe627752b6faec444e1057 not found: ID does not exist"
May 26 11:54:04.621294 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.621241    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"]
May 26 11:54:04.628871 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:04.628848    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-8g2lg"]
May 26 11:54:06.242070 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:06.242001    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" path="/var/lib/kubelet/pods/d268c6e7-03be-49b6-8478-34d39e590ffb/volumes"
May 26 11:54:08.593963 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:08.593932    2563 generic.go:358] "Generic (PLEG): container finished" podID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerID="3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906" exitCode=0
May 26 11:54:08.594359 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:08.593990    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" event={"ID":"ce3d6b1d-a6cd-4353-864b-11b322f5be23","Type":"ContainerDied","Data":"3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906"}
May 26 11:54:09.598400 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:09.598364    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" event={"ID":"ce3d6b1d-a6cd-4353-864b-11b322f5be23","Type":"ContainerStarted","Data":"6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978"}
May 26 11:54:09.598829 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:09.598409    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" event={"ID":"ce3d6b1d-a6cd-4353-864b-11b322f5be23","Type":"ContainerStarted","Data":"ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d"}
May 26 11:54:09.598829 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:09.598621    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:09.619126 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:09.619077    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" podStartSLOduration=6.619062023 podStartE2EDuration="6.619062023s" podCreationTimestamp="2026-05-26 11:54:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:54:09.617204346 +0000 UTC m=+1611.980154015" watchObservedRunningTime="2026-05-26 11:54:09.619062023 +0000 UTC m=+1611.982011691"
May 26 11:54:10.601469 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:10.601434    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:16.610687 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:16.610653    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:46.614414 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:46.614387    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:54.006421 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006391    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"]
May 26 11:54:54.006848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006654    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="storage-initializer"
May 26 11:54:54.006848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="storage-initializer"
May 26 11:54:54.006848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006683    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kube-rbac-proxy"
May 26 11:54:54.006848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006692    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kube-rbac-proxy"
May 26 11:54:54.006848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006713    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kserve-container"
May 26 11:54:54.006848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006719    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kserve-container"
May 26 11:54:54.006848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006757    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kserve-container"
May 26 11:54:54.006848 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.006769    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d268c6e7-03be-49b6-8478-34d39e590ffb" containerName="kube-rbac-proxy"
May 26 11:54:54.008654 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.008637    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.012146 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.012126    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
May 26 11:54:54.012473 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.012447    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
May 26 11:54:54.025085 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.025066    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"]
May 26 11:54:54.026081 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.026056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.026193 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.026099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.026193 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.026129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qss5t\" (UniqueName: \"kubernetes.io/projected/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kube-api-access-qss5t\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.026306 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.026210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d42f-b2b3-4e93-85ad-adc559682e0b-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.050051 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.050022    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"]
May 26 11:54:54.050318 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.050296    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kserve-container" containerID="cri-o://ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d" gracePeriod=30
May 26 11:54:54.050375 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.050327    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kube-rbac-proxy" containerID="cri-o://6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978" gracePeriod=30
May 26 11:54:54.127095 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.126997    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.127282 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:54:54.127120    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 26 11:54:54.127282 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.127132    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.127282 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.127163    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qss5t\" (UniqueName: \"kubernetes.io/projected/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kube-api-access-qss5t\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.127282 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:54:54.127189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls podName:bd97d42f-b2b3-4e93-85ad-adc559682e0b nodeName:}" failed. No retries permitted until 2026-05-26 11:54:54.627168961 +0000 UTC m=+1656.990118621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls") pod "isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" (UID: "bd97d42f-b2b3-4e93-85ad-adc559682e0b") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 26 11:54:54.127282 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.127253    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d42f-b2b3-4e93-85ad-adc559682e0b-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.127582 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.127556    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.127881 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.127861    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d42f-b2b3-4e93-85ad-adc559682e0b-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.138350 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.138322    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qss5t\" (UniqueName: \"kubernetes.io/projected/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kube-api-access-qss5t\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.630346 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.630305    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.632877 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.632854    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:54.718289 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.718255    2563 generic.go:358] "Generic (PLEG): container finished" podID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerID="6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978" exitCode=2
May 26 11:54:54.718416 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.718305    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" event={"ID":"ce3d6b1d-a6cd-4353-864b-11b322f5be23","Type":"ContainerDied","Data":"6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978"}
May 26 11:54:54.917888 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:54.917717    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:54:55.057734 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:55.057704    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"]
May 26 11:54:55.061282 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:54:55.061253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd97d42f_b2b3_4e93_85ad_adc559682e0b.slice/crio-a05dcfb4bab3419fef5287fd9859de9f70f6761213f8b3bf78ca733f10b5803f WatchSource:0}: Error finding container a05dcfb4bab3419fef5287fd9859de9f70f6761213f8b3bf78ca733f10b5803f: Status 404 returned error can't find the container with id a05dcfb4bab3419fef5287fd9859de9f70f6761213f8b3bf78ca733f10b5803f
May 26 11:54:55.722739 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:55.722702    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerStarted","Data":"bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513"}
May 26 11:54:55.722739 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:55.722740    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerStarted","Data":"a05dcfb4bab3419fef5287fd9859de9f70f6761213f8b3bf78ca733f10b5803f"}
May 26 11:54:56.604936 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:56.604892    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 11:54:57.651961 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:57.651920    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-mlflow-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 26 11:54:58.991559 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:58.991534    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:59.061618 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.061593    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdm7m\" (UniqueName: \"kubernetes.io/projected/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kube-api-access-jdm7m\") pod \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") "
May 26 11:54:59.061772 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.061644    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kserve-provision-location\") pod \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") "
May 26 11:54:59.061772 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.061680    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d6b1d-a6cd-4353-864b-11b322f5be23-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") "
May 26 11:54:59.061772 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.061698    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d6b1d-a6cd-4353-864b-11b322f5be23-proxy-tls\") pod \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\" (UID: \"ce3d6b1d-a6cd-4353-864b-11b322f5be23\") "
May 26 11:54:59.062105 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.062070    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce3d6b1d-a6cd-4353-864b-11b322f5be23" (UID: "ce3d6b1d-a6cd-4353-864b-11b322f5be23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:54:59.062202 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.062116    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3d6b1d-a6cd-4353-864b-11b322f5be23-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "ce3d6b1d-a6cd-4353-864b-11b322f5be23" (UID: "ce3d6b1d-a6cd-4353-864b-11b322f5be23"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:54:59.063743 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.063725    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kube-api-access-jdm7m" (OuterVolumeSpecName: "kube-api-access-jdm7m") pod "ce3d6b1d-a6cd-4353-864b-11b322f5be23" (UID: "ce3d6b1d-a6cd-4353-864b-11b322f5be23"). InnerVolumeSpecName "kube-api-access-jdm7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:54:59.063845 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.063804    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3d6b1d-a6cd-4353-864b-11b322f5be23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce3d6b1d-a6cd-4353-864b-11b322f5be23" (UID: "ce3d6b1d-a6cd-4353-864b-11b322f5be23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:54:59.162475 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.162448    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdm7m\" (UniqueName: \"kubernetes.io/projected/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kube-api-access-jdm7m\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:54:59.162475 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.162471    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d6b1d-a6cd-4353-864b-11b322f5be23-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:54:59.162623 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.162482    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d6b1d-a6cd-4353-864b-11b322f5be23-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:54:59.162623 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.162493    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d6b1d-a6cd-4353-864b-11b322f5be23-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:54:59.734830 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.734775    2563 generic.go:358] "Generic (PLEG): container finished" podID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerID="ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d" exitCode=0
May 26 11:54:59.735005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.734824    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" event={"ID":"ce3d6b1d-a6cd-4353-864b-11b322f5be23","Type":"ContainerDied","Data":"ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d"}
May 26 11:54:59.735005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.734866    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws" event={"ID":"ce3d6b1d-a6cd-4353-864b-11b322f5be23","Type":"ContainerDied","Data":"31b8d462e8fdde644d78082a2224a7129cbe787ed27c7ca1a20d7a3c77d98a5e"}
May 26 11:54:59.735005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.734880    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"
May 26 11:54:59.735005 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.734884    2563 scope.go:117] "RemoveContainer" containerID="6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978"
May 26 11:54:59.736314 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.736283    2563 generic.go:358] "Generic (PLEG): container finished" podID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerID="bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513" exitCode=0
May 26 11:54:59.736431 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.736335    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerDied","Data":"bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513"}
May 26 11:54:59.743049 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.743031    2563 scope.go:117] "RemoveContainer" containerID="ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d"
May 26 11:54:59.749899 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.749877    2563 scope.go:117] "RemoveContainer" containerID="3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906"
May 26 11:54:59.756945 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.756927    2563 scope.go:117] "RemoveContainer" containerID="6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978"
May 26 11:54:59.757192 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:54:59.757175    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978\": container with ID starting with 6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978 not found: ID does not exist" containerID="6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978"
May 26 11:54:59.757254 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.757204    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978"} err="failed to get container status \"6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978\": rpc error: code = NotFound desc = could not find container \"6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978\": container with ID starting with 6b0e1241f3bb150586fe7d1a403f0eb3d85cf18ca3e09200cb7adbcf434c3978 not found: ID does not exist"
May 26 11:54:59.757254 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.757227    2563 scope.go:117] "RemoveContainer" containerID="ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d"
May 26 11:54:59.757468 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:54:59.757450    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d\": container with ID starting with ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d not found: ID does not exist" containerID="ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d"
May 26 11:54:59.757517 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.757475    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d"} err="failed to get container status \"ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d\": rpc error: code = NotFound desc = could not find container \"ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d\": container with ID starting with ce2858860bd4c7b8f87ac92b4f340bfb168520249ea780ec6cedb6997b40826d not found: ID does not exist"
May 26 11:54:59.757517 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.757492    2563 scope.go:117] "RemoveContainer" containerID="3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906"
May 26 11:54:59.757714 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:54:59.757697    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906\": container with ID starting with 3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906 not found: ID does not exist" containerID="3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906"
May 26 11:54:59.757756 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.757719    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906"} err="failed to get container status \"3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906\": rpc error: code = NotFound desc = could not find container \"3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906\": container with ID starting with 3011bc066e1c80a17bf077ef2e4b7be9412feb04f9c192ed89faec0de5526906 not found: ID does not exist"
May 26 11:54:59.782809 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.782756    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"]
May 26 11:54:59.786296 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:54:59.786265    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-fm8ws"]
May 26 11:55:00.241899 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:00.241864    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" path="/var/lib/kubelet/pods/ce3d6b1d-a6cd-4353-864b-11b322f5be23/volumes"
May 26 11:55:00.743068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:00.743033    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerStarted","Data":"b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0"}
May 26 11:55:02.751115 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:02.751079    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerStarted","Data":"a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103"}
May 26 11:55:02.751115 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:02.751114    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerStarted","Data":"3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b"}
May 26 11:55:02.751635 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:02.751249    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:55:02.751635 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:02.751351    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:55:02.783329 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:02.783280    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podStartSLOduration=7.443492058 podStartE2EDuration="9.783268178s" podCreationTimestamp="2026-05-26 11:54:53 +0000 UTC" firstStartedPulling="2026-05-26 11:54:59.80782477 +0000 UTC m=+1662.170774417" lastFinishedPulling="2026-05-26 11:55:02.147600876 +0000 UTC m=+1664.510550537" observedRunningTime="2026-05-26 11:55:02.782285792 +0000 UTC m=+1665.145235461" watchObservedRunningTime="2026-05-26 11:55:02.783268178 +0000 UTC m=+1665.146217846"
May 26 11:55:03.754014 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:03.753977    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:55:09.762450 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:09.762420    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:55:29.763068 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:29.763027    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:55:39.764352 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:55:39.764267    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:56:09.764951 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:09.764924    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:56:14.097954 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.097920    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"]
May 26 11:56:14.098481 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.098451    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-container" containerID="cri-o://b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0" gracePeriod=30
May 26 11:56:14.098481 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.098438    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy" containerID="cri-o://a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103" gracePeriod=30
May 26 11:56:14.098703 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.098513    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-agent" containerID="cri-o://3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b" gracePeriod=30
May 26 11:56:14.189740 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.189701    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"]
May 26 11:56:14.190061 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.190044    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kube-rbac-proxy"
May 26 11:56:14.190143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.190063    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kube-rbac-proxy"
May 26 11:56:14.190143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.190079    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kserve-container"
May 26 11:56:14.190143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.190087    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kserve-container"
May 26 11:56:14.190143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.190098    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="storage-initializer"
May 26 11:56:14.190143 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.190106    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="storage-initializer"
May 26 11:56:14.190391 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.190175    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kserve-container"
May 26 11:56:14.190391 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.190188    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce3d6b1d-a6cd-4353-864b-11b322f5be23" containerName="kube-rbac-proxy"
May 26 11:56:14.193328 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.193308    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.196616 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.196594    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
May 26 11:56:14.196721 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.196594    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
May 26 11:56:14.202406 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.202384    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"]
May 26 11:56:14.225952 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.225926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb182f6-816b-415b-93f0-e1331d9a68bc-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.226070 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.225965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb182f6-816b-415b-93f0-e1331d9a68bc-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.226070 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.226001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb182f6-816b-415b-93f0-e1331d9a68bc-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.226070 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.226023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gh5\" (UniqueName: \"kubernetes.io/projected/aeb182f6-816b-415b-93f0-e1331d9a68bc-kube-api-access-n8gh5\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.326498 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.326461    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb182f6-816b-415b-93f0-e1331d9a68bc-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.326670 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.326504    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb182f6-816b-415b-93f0-e1331d9a68bc-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.326670 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.326542    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb182f6-816b-415b-93f0-e1331d9a68bc-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.326670 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.326564    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gh5\" (UniqueName: \"kubernetes.io/projected/aeb182f6-816b-415b-93f0-e1331d9a68bc-kube-api-access-n8gh5\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.327040 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.327018    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb182f6-816b-415b-93f0-e1331d9a68bc-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.327249 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.327229    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb182f6-816b-415b-93f0-e1331d9a68bc-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.329171 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.329153    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb182f6-816b-415b-93f0-e1331d9a68bc-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.335023 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.335002    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gh5\" (UniqueName: \"kubernetes.io/projected/aeb182f6-816b-415b-93f0-e1331d9a68bc-kube-api-access-n8gh5\") pod \"isvc-paddle-predictor-6b8b7cfb4b-nzzhv\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.503853 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.503754    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:14.636420 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.636319    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"]
May 26 11:56:14.638884 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:56:14.638861    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb182f6_816b_415b_93f0_e1331d9a68bc.slice/crio-d9996b69cf601306acd132741cf0c44958b87b9cb26978430398266db80edfe2 WatchSource:0}: Error finding container d9996b69cf601306acd132741cf0c44958b87b9cb26978430398266db80edfe2: Status 404 returned error can't find the container with id d9996b69cf601306acd132741cf0c44958b87b9cb26978430398266db80edfe2
May 26 11:56:14.758145 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.758054    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
May 26 11:56:14.957958 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.957913    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" event={"ID":"aeb182f6-816b-415b-93f0-e1331d9a68bc","Type":"ContainerStarted","Data":"faad480e9fd3789ffd6b6d1f4f7c47ae38f338020f9053175b64f7796ae90a74"}
May 26 11:56:14.957958 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.957960    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" event={"ID":"aeb182f6-816b-415b-93f0-e1331d9a68bc","Type":"ContainerStarted","Data":"d9996b69cf601306acd132741cf0c44958b87b9cb26978430398266db80edfe2"}
May 26 11:56:14.960219 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.960193    2563 generic.go:358] "Generic (PLEG): container finished" podID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerID="a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103" exitCode=2
May 26 11:56:14.960365 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:14.960236    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerDied","Data":"a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103"}
May 26 11:56:16.967440 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:16.967409    2563 generic.go:358] "Generic (PLEG): container finished" podID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerID="b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0" exitCode=0
May 26 11:56:16.967860 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:16.967462    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerDied","Data":"b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0"}
May 26 11:56:19.757730 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:19.757685    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
May 26 11:56:19.763581 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:19.763549    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:56:19.975557 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:19.975523    2563 generic.go:358] "Generic (PLEG): container finished" podID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerID="faad480e9fd3789ffd6b6d1f4f7c47ae38f338020f9053175b64f7796ae90a74" exitCode=0
May 26 11:56:19.975724 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:19.975574    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" event={"ID":"aeb182f6-816b-415b-93f0-e1331d9a68bc","Type":"ContainerDied","Data":"faad480e9fd3789ffd6b6d1f4f7c47ae38f338020f9053175b64f7796ae90a74"}
May 26 11:56:24.758047 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:24.758002    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
May 26 11:56:24.758483 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:24.758156    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:56:29.758159 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:29.758112    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
May 26 11:56:29.763737 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:29.763705    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:56:32.011079 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:32.011041    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" event={"ID":"aeb182f6-816b-415b-93f0-e1331d9a68bc","Type":"ContainerStarted","Data":"4a58c594187c7f7d039b515ea78f1131cc9d3bed43e3ead4dfbd0b8d875af55f"}
May 26 11:56:32.011079 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:32.011083    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" event={"ID":"aeb182f6-816b-415b-93f0-e1331d9a68bc","Type":"ContainerStarted","Data":"63a31f53042d04776a143436813fd59761fd6330d5c497232f00afcd9cc3c8e5"}
May 26 11:56:32.011486 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:32.011285    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:32.035112 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:32.035069    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podStartSLOduration=6.608759659 podStartE2EDuration="18.035056373s" podCreationTimestamp="2026-05-26 11:56:14 +0000 UTC" firstStartedPulling="2026-05-26 11:56:19.976691666 +0000 UTC m=+1742.339641313" lastFinishedPulling="2026-05-26 11:56:31.402988378 +0000 UTC m=+1753.765938027" observedRunningTime="2026-05-26 11:56:32.0323869 +0000 UTC m=+1754.395336569" watchObservedRunningTime="2026-05-26 11:56:32.035056373 +0000 UTC m=+1754.398006043"
May 26 11:56:33.013622 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:33.013594    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:33.014831 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:33.014788    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 11:56:34.016229 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:34.016188    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 11:56:34.757288 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:34.757242    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
May 26 11:56:39.020634 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:39.020603    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:56:39.021128 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:39.021105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 11:56:39.757877 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:39.757837    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
May 26 11:56:39.763208 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:39.763183    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:56:39.763299 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:39.763288    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:56:44.240328 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.240290    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:56:44.360082 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.359994    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls\") pod \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") "
May 26 11:56:44.360082 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.360055    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qss5t\" (UniqueName: \"kubernetes.io/projected/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kube-api-access-qss5t\") pod \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") "
May 26 11:56:44.360272 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.360087    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d42f-b2b3-4e93-85ad-adc559682e0b-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") "
May 26 11:56:44.360272 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.360129    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kserve-provision-location\") pod \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\" (UID: \"bd97d42f-b2b3-4e93-85ad-adc559682e0b\") "
May 26 11:56:44.360529 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.360492    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd97d42f-b2b3-4e93-85ad-adc559682e0b" (UID: "bd97d42f-b2b3-4e93-85ad-adc559682e0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:56:44.360529 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.360508    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd97d42f-b2b3-4e93-85ad-adc559682e0b-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "bd97d42f-b2b3-4e93-85ad-adc559682e0b" (UID: "bd97d42f-b2b3-4e93-85ad-adc559682e0b"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:56:44.362241 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.362222    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd97d42f-b2b3-4e93-85ad-adc559682e0b" (UID: "bd97d42f-b2b3-4e93-85ad-adc559682e0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:56:44.362350 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.362331    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kube-api-access-qss5t" (OuterVolumeSpecName: "kube-api-access-qss5t") pod "bd97d42f-b2b3-4e93-85ad-adc559682e0b" (UID: "bd97d42f-b2b3-4e93-85ad-adc559682e0b"). InnerVolumeSpecName "kube-api-access-qss5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:56:44.460712 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.460681    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d42f-b2b3-4e93-85ad-adc559682e0b-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:56:44.460712 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.460707    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:56:44.460712 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.460717    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d42f-b2b3-4e93-85ad-adc559682e0b-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:56:44.460934 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:44.460726    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qss5t\" (UniqueName: \"kubernetes.io/projected/bd97d42f-b2b3-4e93-85ad-adc559682e0b-kube-api-access-qss5t\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:56:45.049291 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.049261    2563 generic.go:358] "Generic (PLEG): container finished" podID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerID="3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b" exitCode=137
May 26 11:56:45.049453 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.049356    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"
May 26 11:56:45.049453 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.049355    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerDied","Data":"3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b"}
May 26 11:56:45.049453 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.049402    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws" event={"ID":"bd97d42f-b2b3-4e93-85ad-adc559682e0b","Type":"ContainerDied","Data":"a05dcfb4bab3419fef5287fd9859de9f70f6761213f8b3bf78ca733f10b5803f"}
May 26 11:56:45.049453 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.049424    2563 scope.go:117] "RemoveContainer" containerID="a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103"
May 26 11:56:45.057711 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.057684    2563 scope.go:117] "RemoveContainer" containerID="3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b"
May 26 11:56:45.064559 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.064542    2563 scope.go:117] "RemoveContainer" containerID="b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0"
May 26 11:56:45.070853 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.070830    2563 scope.go:117] "RemoveContainer" containerID="bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513"
May 26 11:56:45.075670 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.075647    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"]
May 26 11:56:45.078037 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.077909    2563 scope.go:117] "RemoveContainer" containerID="a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103"
May 26 11:56:45.078240 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:56:45.078214    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103\": container with ID starting with a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103 not found: ID does not exist" containerID="a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103"
May 26 11:56:45.078304 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.078249    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103"} err="failed to get container status \"a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103\": rpc error: code = NotFound desc = could not find container \"a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103\": container with ID starting with a6976fadfe3810c0206508f491b3182d8cf7665c551795e77d775c87d7cc2103 not found: ID does not exist"
May 26 11:56:45.078304 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.078273    2563 scope.go:117] "RemoveContainer" containerID="3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b"
May 26 11:56:45.078560 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:56:45.078541    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b\": container with ID starting with 3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b not found: ID does not exist" containerID="3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b"
May 26 11:56:45.078633 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.078570    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b"} err="failed to get container status \"3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b\": rpc error: code = NotFound desc = could not find container \"3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b\": container with ID starting with 3b2966004594f3719d791f1e20eab02b09acbdfb4f8c0ed934fe7fceb76b860b not found: ID does not exist"
May 26 11:56:45.078633 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.078593    2563 scope.go:117] "RemoveContainer" containerID="b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0"
May 26 11:56:45.078899 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:56:45.078870    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0\": container with ID starting with b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0 not found: ID does not exist" containerID="b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0"
May 26 11:56:45.078965 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.078906    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0"} err="failed to get container status \"b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0\": rpc error: code = NotFound desc = could not find container \"b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0\": container with ID starting with b9945650e58933fd27c90748b70f211fa0b99c21640fb5547bf6db3c43f0f3d0 not found: ID does not exist"
May 26 11:56:45.078965 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.078923    2563 scope.go:117] "RemoveContainer" containerID="bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513"
May 26 11:56:45.079177 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:56:45.079159    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513\": container with ID starting with bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513 not found: ID does not exist" containerID="bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513"
May 26 11:56:45.079220 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.079185    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513"} err="failed to get container status \"bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513\": rpc error: code = NotFound desc = could not find container \"bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513\": container with ID starting with bf96ed6bbae4093ffc9d73485bb029519cb20202c1538bef1475484568593513 not found: ID does not exist"
May 26 11:56:45.079643 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:45.079624    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-9c5ws"]
May 26 11:56:46.242714 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:46.242674    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" path="/var/lib/kubelet/pods/bd97d42f-b2b3-4e93-85ad-adc559682e0b/volumes"
May 26 11:56:49.021214 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:49.021171    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 11:56:59.021221 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:56:59.021181    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 11:57:09.021774 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:09.021692    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 11:57:19.022598 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:19.022568    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:57:25.628141 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.628109    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"]
May 26 11:57:25.628600 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.628429    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container" containerID="cri-o://63a31f53042d04776a143436813fd59761fd6330d5c497232f00afcd9cc3c8e5" gracePeriod=30
May 26 11:57:25.628600 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.628475    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kube-rbac-proxy" containerID="cri-o://4a58c594187c7f7d039b515ea78f1131cc9d3bed43e3ead4dfbd0b8d875af55f" gracePeriod=30
May 26 11:57:25.686626 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686594    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"]
May 26 11:57:25.686861 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686849    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-agent"
May 26 11:57:25.686910 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686862    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-agent"
May 26 11:57:25.686910 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="storage-initializer"
May 26 11:57:25.686910 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686884    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="storage-initializer"
May 26 11:57:25.686910 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686893    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-container"
May 26 11:57:25.686910 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686899    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-container"
May 26 11:57:25.686910 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686908    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy"
May 26 11:57:25.687108 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy"
May 26 11:57:25.687108 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686953    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-container"
May 26 11:57:25.687108 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686964    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kube-rbac-proxy"
May 26 11:57:25.687108 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.686971    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="bd97d42f-b2b3-4e93-85ad-adc559682e0b" containerName="kserve-agent"
May 26 11:57:25.690965 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.690943    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.693417 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.693396    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
May 26 11:57:25.694210 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.694194    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
May 26 11:57:25.703906 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.703884    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"]
May 26 11:57:25.765725 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.765702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tngk2\" (UniqueName: \"kubernetes.io/projected/5520e035-56bf-4d0d-a40a-bfb52f25759b-kube-api-access-tngk2\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.765857 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.765734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5520e035-56bf-4d0d-a40a-bfb52f25759b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.765857 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.765756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5520e035-56bf-4d0d-a40a-bfb52f25759b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.765953 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.765858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5520e035-56bf-4d0d-a40a-bfb52f25759b-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.866372 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.866334    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5520e035-56bf-4d0d-a40a-bfb52f25759b-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.866543 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.866399    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tngk2\" (UniqueName: \"kubernetes.io/projected/5520e035-56bf-4d0d-a40a-bfb52f25759b-kube-api-access-tngk2\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.866543 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.866439    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5520e035-56bf-4d0d-a40a-bfb52f25759b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.866543 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.866484    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5520e035-56bf-4d0d-a40a-bfb52f25759b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.866893 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.866868    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5520e035-56bf-4d0d-a40a-bfb52f25759b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.867108 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.867078    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5520e035-56bf-4d0d-a40a-bfb52f25759b-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.869104 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.869084    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5520e035-56bf-4d0d-a40a-bfb52f25759b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:25.874685 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:25.874663    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngk2\" (UniqueName: \"kubernetes.io/projected/5520e035-56bf-4d0d-a40a-bfb52f25759b-kube-api-access-tngk2\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:26.001775 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:26.001683    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:26.128686 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:26.128644    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"]
May 26 11:57:26.131589 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:57:26.131563    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5520e035_56bf_4d0d_a40a_bfb52f25759b.slice/crio-f4ac59a19134dc4cb9d8eecb7c50da6c414d9a5b207592bf572487020b4781db WatchSource:0}: Error finding container f4ac59a19134dc4cb9d8eecb7c50da6c414d9a5b207592bf572487020b4781db: Status 404 returned error can't find the container with id f4ac59a19134dc4cb9d8eecb7c50da6c414d9a5b207592bf572487020b4781db
May 26 11:57:26.159158 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:26.159132    2563 generic.go:358] "Generic (PLEG): container finished" podID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerID="4a58c594187c7f7d039b515ea78f1131cc9d3bed43e3ead4dfbd0b8d875af55f" exitCode=2
May 26 11:57:26.159257 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:26.159200    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" event={"ID":"aeb182f6-816b-415b-93f0-e1331d9a68bc","Type":"ContainerDied","Data":"4a58c594187c7f7d039b515ea78f1131cc9d3bed43e3ead4dfbd0b8d875af55f"}
May 26 11:57:26.160204 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:26.160181    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" event={"ID":"5520e035-56bf-4d0d-a40a-bfb52f25759b","Type":"ContainerStarted","Data":"f4ac59a19134dc4cb9d8eecb7c50da6c414d9a5b207592bf572487020b4781db"}
May 26 11:57:27.164149 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:27.164112    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" event={"ID":"5520e035-56bf-4d0d-a40a-bfb52f25759b","Type":"ContainerStarted","Data":"01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227"}
May 26 11:57:28.169038 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.169002    2563 generic.go:358] "Generic (PLEG): container finished" podID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerID="63a31f53042d04776a143436813fd59761fd6330d5c497232f00afcd9cc3c8e5" exitCode=0
May 26 11:57:28.169366 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.169070    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" event={"ID":"aeb182f6-816b-415b-93f0-e1331d9a68bc","Type":"ContainerDied","Data":"63a31f53042d04776a143436813fd59761fd6330d5c497232f00afcd9cc3c8e5"}
May 26 11:57:28.268372 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.268343    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:57:28.282703 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.282683    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb182f6-816b-415b-93f0-e1331d9a68bc-proxy-tls\") pod \"aeb182f6-816b-415b-93f0-e1331d9a68bc\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") "
May 26 11:57:28.282785 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.282721    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8gh5\" (UniqueName: \"kubernetes.io/projected/aeb182f6-816b-415b-93f0-e1331d9a68bc-kube-api-access-n8gh5\") pod \"aeb182f6-816b-415b-93f0-e1331d9a68bc\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") "
May 26 11:57:28.282839 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.282787    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb182f6-816b-415b-93f0-e1331d9a68bc-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"aeb182f6-816b-415b-93f0-e1331d9a68bc\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") "
May 26 11:57:28.282870 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.282846    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb182f6-816b-415b-93f0-e1331d9a68bc-kserve-provision-location\") pod \"aeb182f6-816b-415b-93f0-e1331d9a68bc\" (UID: \"aeb182f6-816b-415b-93f0-e1331d9a68bc\") "
May 26 11:57:28.283191 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.283168    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb182f6-816b-415b-93f0-e1331d9a68bc-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "aeb182f6-816b-415b-93f0-e1331d9a68bc" (UID: "aeb182f6-816b-415b-93f0-e1331d9a68bc"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:57:28.284829 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.284808    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb182f6-816b-415b-93f0-e1331d9a68bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aeb182f6-816b-415b-93f0-e1331d9a68bc" (UID: "aeb182f6-816b-415b-93f0-e1331d9a68bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:57:28.284991 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.284969    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb182f6-816b-415b-93f0-e1331d9a68bc-kube-api-access-n8gh5" (OuterVolumeSpecName: "kube-api-access-n8gh5") pod "aeb182f6-816b-415b-93f0-e1331d9a68bc" (UID: "aeb182f6-816b-415b-93f0-e1331d9a68bc"). InnerVolumeSpecName "kube-api-access-n8gh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:57:28.293694 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.293664    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb182f6-816b-415b-93f0-e1331d9a68bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aeb182f6-816b-415b-93f0-e1331d9a68bc" (UID: "aeb182f6-816b-415b-93f0-e1331d9a68bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:57:28.383396 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.383287    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8gh5\" (UniqueName: \"kubernetes.io/projected/aeb182f6-816b-415b-93f0-e1331d9a68bc-kube-api-access-n8gh5\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:57:28.383396 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.383336    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb182f6-816b-415b-93f0-e1331d9a68bc-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:57:28.383396 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.383350    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb182f6-816b-415b-93f0-e1331d9a68bc-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:57:28.383396 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:28.383362    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb182f6-816b-415b-93f0-e1331d9a68bc-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:57:29.172816 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:29.172766    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv" event={"ID":"aeb182f6-816b-415b-93f0-e1331d9a68bc","Type":"ContainerDied","Data":"d9996b69cf601306acd132741cf0c44958b87b9cb26978430398266db80edfe2"}
May 26 11:57:29.173278 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:29.172839    2563 scope.go:117] "RemoveContainer" containerID="4a58c594187c7f7d039b515ea78f1131cc9d3bed43e3ead4dfbd0b8d875af55f"
May 26 11:57:29.173278 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:29.172862    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"
May 26 11:57:29.180825 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:29.180784    2563 scope.go:117] "RemoveContainer" containerID="63a31f53042d04776a143436813fd59761fd6330d5c497232f00afcd9cc3c8e5"
May 26 11:57:29.187609 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:29.187592    2563 scope.go:117] "RemoveContainer" containerID="faad480e9fd3789ffd6b6d1f4f7c47ae38f338020f9053175b64f7796ae90a74"
May 26 11:57:29.194280 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:29.194199    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"]
May 26 11:57:29.197975 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:29.197952    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-nzzhv"]
May 26 11:57:30.242597 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:30.242566    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" path="/var/lib/kubelet/pods/aeb182f6-816b-415b-93f0-e1331d9a68bc/volumes"
May 26 11:57:31.180907 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:31.180874    2563 generic.go:358] "Generic (PLEG): container finished" podID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerID="01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227" exitCode=0
May 26 11:57:31.181087 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:31.180921    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" event={"ID":"5520e035-56bf-4d0d-a40a-bfb52f25759b","Type":"ContainerDied","Data":"01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227"}
May 26 11:57:32.184927 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:32.184888    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" event={"ID":"5520e035-56bf-4d0d-a40a-bfb52f25759b","Type":"ContainerStarted","Data":"a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647"}
May 26 11:57:32.185307 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:32.184936    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" event={"ID":"5520e035-56bf-4d0d-a40a-bfb52f25759b","Type":"ContainerStarted","Data":"69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d"}
May 26 11:57:32.185307 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:32.185158    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:32.203386 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:32.203345    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podStartSLOduration=7.203332503 podStartE2EDuration="7.203332503s" podCreationTimestamp="2026-05-26 11:57:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:57:32.202883226 +0000 UTC m=+1814.565832896" watchObservedRunningTime="2026-05-26 11:57:32.203332503 +0000 UTC m=+1814.566282172"
May 26 11:57:33.188166 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:33.188135    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:33.189213 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:33.189189    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 26 11:57:34.191046 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:34.191006    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 26 11:57:39.195231 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:39.195204    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:57:39.195736 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:39.195711    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 26 11:57:49.196427 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:49.196384    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 26 11:57:59.196406 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:57:59.196364    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 26 11:58:01.362082 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:01.362052    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:58:01.363596 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:01.363573    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 11:58:09.195739 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:09.195700    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 26 11:58:19.196947 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:19.196917    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:58:27.095821 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.095775    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"]
May 26 11:58:27.096364 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.096119    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container" containerID="cri-o://69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d" gracePeriod=30
May 26 11:58:27.096364 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.096138    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kube-rbac-proxy" containerID="cri-o://a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647" gracePeriod=30
May 26 11:58:27.161921 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.161885    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"]
May 26 11:58:27.162222 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.162204    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="storage-initializer"
May 26 11:58:27.162303 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.162224    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="storage-initializer"
May 26 11:58:27.162303 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.162236    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kube-rbac-proxy"
May 26 11:58:27.162303 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.162244    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kube-rbac-proxy"
May 26 11:58:27.162303 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.162266    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container"
May 26 11:58:27.162303 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.162274    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container"
May 26 11:58:27.162544 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.162346    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kserve-container"
May 26 11:58:27.162544 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.162358    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="aeb182f6-816b-415b-93f0-e1331d9a68bc" containerName="kube-rbac-proxy"
May 26 11:58:27.166343 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.166323    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.168599 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.168571    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
May 26 11:58:27.168710 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.168629    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
May 26 11:58:27.176657 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.176637    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"]
May 26 11:58:27.303277 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.303239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdvq\" (UniqueName: \"kubernetes.io/projected/732ed8a4-6885-479f-b579-4359355cc8ea-kube-api-access-5zdvq\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.303436 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.303304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732ed8a4-6885-479f-b579-4359355cc8ea-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.303436 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.303338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732ed8a4-6885-479f-b579-4359355cc8ea-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.303436 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.303408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732ed8a4-6885-479f-b579-4359355cc8ea-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.331851 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.331820    2563 generic.go:358] "Generic (PLEG): container finished" podID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerID="a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647" exitCode=2
May 26 11:58:27.331974 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.331888    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" event={"ID":"5520e035-56bf-4d0d-a40a-bfb52f25759b","Type":"ContainerDied","Data":"a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647"}
May 26 11:58:27.404164 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.404079    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdvq\" (UniqueName: \"kubernetes.io/projected/732ed8a4-6885-479f-b579-4359355cc8ea-kube-api-access-5zdvq\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.404164 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.404134    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732ed8a4-6885-479f-b579-4359355cc8ea-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.404364 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.404164    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732ed8a4-6885-479f-b579-4359355cc8ea-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.404364 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.404207    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732ed8a4-6885-479f-b579-4359355cc8ea-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.404569 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.404549    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732ed8a4-6885-479f-b579-4359355cc8ea-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.405004 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.404979    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732ed8a4-6885-479f-b579-4359355cc8ea-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.406930 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.406909    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732ed8a4-6885-479f-b579-4359355cc8ea-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.411924 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.411902    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdvq\" (UniqueName: \"kubernetes.io/projected/732ed8a4-6885-479f-b579-4359355cc8ea-kube-api-access-5zdvq\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.476331 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.476300    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:27.599595 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.599556    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"]
May 26 11:58:27.602813 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:58:27.602766    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732ed8a4_6885_479f_b579_4359355cc8ea.slice/crio-b2b8d403e938aa6e3f7b534fce0745e27a4c581985044b522f90476b231cb020 WatchSource:0}: Error finding container b2b8d403e938aa6e3f7b534fce0745e27a4c581985044b522f90476b231cb020: Status 404 returned error can't find the container with id b2b8d403e938aa6e3f7b534fce0745e27a4c581985044b522f90476b231cb020
May 26 11:58:27.604626 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:27.604606    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:58:28.338258 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:28.338224    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" event={"ID":"732ed8a4-6885-479f-b579-4359355cc8ea","Type":"ContainerStarted","Data":"9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883"}
May 26 11:58:28.338258 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:28.338259    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" event={"ID":"732ed8a4-6885-479f-b579-4359355cc8ea","Type":"ContainerStarted","Data":"b2b8d403e938aa6e3f7b534fce0745e27a4c581985044b522f90476b231cb020"}
May 26 11:58:29.191438 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.191395    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
May 26 11:58:29.195808 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.195757    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
May 26 11:58:29.735810 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.735776    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:58:29.823965 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.823938    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tngk2\" (UniqueName: \"kubernetes.io/projected/5520e035-56bf-4d0d-a40a-bfb52f25759b-kube-api-access-tngk2\") pod \"5520e035-56bf-4d0d-a40a-bfb52f25759b\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") "
May 26 11:58:29.824133 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.823979    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5520e035-56bf-4d0d-a40a-bfb52f25759b-kserve-provision-location\") pod \"5520e035-56bf-4d0d-a40a-bfb52f25759b\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") "
May 26 11:58:29.824133 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.824000    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5520e035-56bf-4d0d-a40a-bfb52f25759b-proxy-tls\") pod \"5520e035-56bf-4d0d-a40a-bfb52f25759b\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") "
May 26 11:58:29.824133 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.824035    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5520e035-56bf-4d0d-a40a-bfb52f25759b-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"5520e035-56bf-4d0d-a40a-bfb52f25759b\" (UID: \"5520e035-56bf-4d0d-a40a-bfb52f25759b\") "
May 26 11:58:29.824449 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.824411    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5520e035-56bf-4d0d-a40a-bfb52f25759b-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "5520e035-56bf-4d0d-a40a-bfb52f25759b" (UID: "5520e035-56bf-4d0d-a40a-bfb52f25759b"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:58:29.826179 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.826148    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5520e035-56bf-4d0d-a40a-bfb52f25759b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5520e035-56bf-4d0d-a40a-bfb52f25759b" (UID: "5520e035-56bf-4d0d-a40a-bfb52f25759b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:58:29.826271 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.826229    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5520e035-56bf-4d0d-a40a-bfb52f25759b-kube-api-access-tngk2" (OuterVolumeSpecName: "kube-api-access-tngk2") pod "5520e035-56bf-4d0d-a40a-bfb52f25759b" (UID: "5520e035-56bf-4d0d-a40a-bfb52f25759b"). InnerVolumeSpecName "kube-api-access-tngk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:58:29.833932 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.833907    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5520e035-56bf-4d0d-a40a-bfb52f25759b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5520e035-56bf-4d0d-a40a-bfb52f25759b" (UID: "5520e035-56bf-4d0d-a40a-bfb52f25759b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:58:29.925181 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.925154    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tngk2\" (UniqueName: \"kubernetes.io/projected/5520e035-56bf-4d0d-a40a-bfb52f25759b-kube-api-access-tngk2\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:58:29.925181 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.925176    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5520e035-56bf-4d0d-a40a-bfb52f25759b-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:58:29.925376 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.925187    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5520e035-56bf-4d0d-a40a-bfb52f25759b-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:58:29.925376 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:29.925197    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5520e035-56bf-4d0d-a40a-bfb52f25759b-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:58:30.344909 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.344871    2563 generic.go:358] "Generic (PLEG): container finished" podID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerID="69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d" exitCode=0
May 26 11:58:30.345091 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.344965    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"
May 26 11:58:30.345091 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.344960    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" event={"ID":"5520e035-56bf-4d0d-a40a-bfb52f25759b","Type":"ContainerDied","Data":"69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d"}
May 26 11:58:30.345091 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.345007    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9" event={"ID":"5520e035-56bf-4d0d-a40a-bfb52f25759b","Type":"ContainerDied","Data":"f4ac59a19134dc4cb9d8eecb7c50da6c414d9a5b207592bf572487020b4781db"}
May 26 11:58:30.345091 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.345029    2563 scope.go:117] "RemoveContainer" containerID="a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647"
May 26 11:58:30.352411 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.352395    2563 scope.go:117] "RemoveContainer" containerID="69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d"
May 26 11:58:30.358958 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.358939    2563 scope.go:117] "RemoveContainer" containerID="01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227"
May 26 11:58:30.361161 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.361139    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"]
May 26 11:58:30.365040 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.365012    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9m7f9"]
May 26 11:58:30.366337 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.366314    2563 scope.go:117] "RemoveContainer" containerID="a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647"
May 26 11:58:30.366576 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:58:30.366558    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647\": container with ID starting with a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647 not found: ID does not exist" containerID="a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647"
May 26 11:58:30.366628 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.366587    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647"} err="failed to get container status \"a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647\": rpc error: code = NotFound desc = could not find container \"a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647\": container with ID starting with a663ebf2a6be59bec75eef24aaa745dc055a6751072847a016e1822b24bfe647 not found: ID does not exist"
May 26 11:58:30.366628 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.366607    2563 scope.go:117] "RemoveContainer" containerID="69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d"
May 26 11:58:30.366847 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:58:30.366828    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d\": container with ID starting with 69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d not found: ID does not exist" containerID="69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d"
May 26 11:58:30.366918 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.366856    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d"} err="failed to get container status \"69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d\": rpc error: code = NotFound desc = could not find container \"69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d\": container with ID starting with 69903b2e5816bfb8249d13f6bec3040c1df666818d29b634a7fb6d20119a729d not found: ID does not exist"
May 26 11:58:30.366918 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.366879    2563 scope.go:117] "RemoveContainer" containerID="01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227"
May 26 11:58:30.367117 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:58:30.367101    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227\": container with ID starting with 01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227 not found: ID does not exist" containerID="01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227"
May 26 11:58:30.367159 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:30.367121    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227"} err="failed to get container status \"01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227\": rpc error: code = NotFound desc = could not find container \"01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227\": container with ID starting with 01989a6ad44ba13c6f821e24dd7e0794bd33765646f00e032ab9fc25e8053227 not found: ID does not exist"
May 26 11:58:32.242754 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:32.242722    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" path="/var/lib/kubelet/pods/5520e035-56bf-4d0d-a40a-bfb52f25759b/volumes"
May 26 11:58:32.353289 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:32.353260    2563 generic.go:358] "Generic (PLEG): container finished" podID="732ed8a4-6885-479f-b579-4359355cc8ea" containerID="9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883" exitCode=0
May 26 11:58:32.353451 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:32.353315    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" event={"ID":"732ed8a4-6885-479f-b579-4359355cc8ea","Type":"ContainerDied","Data":"9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883"}
May 26 11:58:33.357199 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:33.357164    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" event={"ID":"732ed8a4-6885-479f-b579-4359355cc8ea","Type":"ContainerStarted","Data":"43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707"}
May 26 11:58:33.357199 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:33.357203    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" event={"ID":"732ed8a4-6885-479f-b579-4359355cc8ea","Type":"ContainerStarted","Data":"c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a"}
May 26 11:58:33.357592 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:33.357523    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:33.357592 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:33.357552    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:33.358673 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:33.358650    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:58:33.376092 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:33.376043    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podStartSLOduration=6.376031265 podStartE2EDuration="6.376031265s" podCreationTimestamp="2026-05-26 11:58:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:58:33.375321022 +0000 UTC m=+1875.738270696" watchObservedRunningTime="2026-05-26 11:58:33.376031265 +0000 UTC m=+1875.738980934"
May 26 11:58:34.360494 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:34.360452    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:58:39.365139 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:39.365064    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:58:39.365608 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:39.365583    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:58:49.365777 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:49.365736    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:58:59.366145 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:58:59.366105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:59:09.365806 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:09.365764    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:59:19.366827 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:19.366775    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:59:28.763403 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.763370    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"]
May 26 11:59:28.763779 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.763691    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container" containerID="cri-o://c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a" gracePeriod=30
May 26 11:59:28.763779 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.763713    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kube-rbac-proxy" containerID="cri-o://43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707" gracePeriod=30
May 26 11:59:28.960601 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960565    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"]
May 26 11:59:28.960862 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960849    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container"
May 26 11:59:28.960913 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960864    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container"
May 26 11:59:28.960913 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960876    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="storage-initializer"
May 26 11:59:28.960913 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960882    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="storage-initializer"
May 26 11:59:28.960913 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960889    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kube-rbac-proxy"
May 26 11:59:28.960913 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960895    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kube-rbac-proxy"
May 26 11:59:28.961061 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960953    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kube-rbac-proxy"
May 26 11:59:28.961061 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.960960    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="5520e035-56bf-4d0d-a40a-bfb52f25759b" containerName="kserve-container"
May 26 11:59:28.963900 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.963884    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:28.966699 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.966669    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
May 26 11:59:28.966904 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.966877    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
May 26 11:59:28.979711 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:28.979688    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"]
May 26 11:59:29.041211 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.041186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe57b3b-b3bf-44b5-94fd-69082450e246-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.041349 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.041227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8xp\" (UniqueName: \"kubernetes.io/projected/5fe57b3b-b3bf-44b5-94fd-69082450e246-kube-api-access-5l8xp\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.041349 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.041247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe57b3b-b3bf-44b5-94fd-69082450e246-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.041419 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.041341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe57b3b-b3bf-44b5-94fd-69082450e246-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.141648 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.141617    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe57b3b-b3bf-44b5-94fd-69082450e246-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.141832 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.141665    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe57b3b-b3bf-44b5-94fd-69082450e246-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.141832 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.141697    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8xp\" (UniqueName: \"kubernetes.io/projected/5fe57b3b-b3bf-44b5-94fd-69082450e246-kube-api-access-5l8xp\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.141832 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.141715    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe57b3b-b3bf-44b5-94fd-69082450e246-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.142159 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.142134    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe57b3b-b3bf-44b5-94fd-69082450e246-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.142337 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.142309    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe57b3b-b3bf-44b5-94fd-69082450e246-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.144309 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.144286    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe57b3b-b3bf-44b5-94fd-69082450e246-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.150330 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.150309    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8xp\" (UniqueName: \"kubernetes.io/projected/5fe57b3b-b3bf-44b5-94fd-69082450e246-kube-api-access-5l8xp\") pod \"isvc-pmml-predictor-8bb578669-ldf5f\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.273651 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.273614    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:29.361319 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.361284    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 26 11:59:29.366565 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.366525    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:59:29.401745 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.401721    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"]
May 26 11:59:29.404452 ip-10-0-138-207 kubenswrapper[2563]: W0526 11:59:29.404422    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe57b3b_b3bf_44b5_94fd_69082450e246.slice/crio-a16494a972e8d3dbf0463e33cd31890041f6d138dd7212cfa7a178d9f4514a19 WatchSource:0}: Error finding container a16494a972e8d3dbf0463e33cd31890041f6d138dd7212cfa7a178d9f4514a19: Status 404 returned error can't find the container with id a16494a972e8d3dbf0463e33cd31890041f6d138dd7212cfa7a178d9f4514a19
May 26 11:59:29.509159 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.509115    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" event={"ID":"5fe57b3b-b3bf-44b5-94fd-69082450e246","Type":"ContainerStarted","Data":"b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1"}
May 26 11:59:29.509159 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.509163    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" event={"ID":"5fe57b3b-b3bf-44b5-94fd-69082450e246","Type":"ContainerStarted","Data":"a16494a972e8d3dbf0463e33cd31890041f6d138dd7212cfa7a178d9f4514a19"}
May 26 11:59:29.511224 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.511191    2563 generic.go:358] "Generic (PLEG): container finished" podID="732ed8a4-6885-479f-b579-4359355cc8ea" containerID="43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707" exitCode=2
May 26 11:59:29.511312 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:29.511230    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" event={"ID":"732ed8a4-6885-479f-b579-4359355cc8ea","Type":"ContainerDied","Data":"43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707"}
May 26 11:59:31.411414 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.411386    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:59:31.460370 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.460337    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zdvq\" (UniqueName: \"kubernetes.io/projected/732ed8a4-6885-479f-b579-4359355cc8ea-kube-api-access-5zdvq\") pod \"732ed8a4-6885-479f-b579-4359355cc8ea\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") "
May 26 11:59:31.460515 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.460379    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732ed8a4-6885-479f-b579-4359355cc8ea-kserve-provision-location\") pod \"732ed8a4-6885-479f-b579-4359355cc8ea\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") "
May 26 11:59:31.460515 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.460417    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732ed8a4-6885-479f-b579-4359355cc8ea-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"732ed8a4-6885-479f-b579-4359355cc8ea\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") "
May 26 11:59:31.460515 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.460477    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732ed8a4-6885-479f-b579-4359355cc8ea-proxy-tls\") pod \"732ed8a4-6885-479f-b579-4359355cc8ea\" (UID: \"732ed8a4-6885-479f-b579-4359355cc8ea\") "
May 26 11:59:31.460697 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.460682    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732ed8a4-6885-479f-b579-4359355cc8ea-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "732ed8a4-6885-479f-b579-4359355cc8ea" (UID: "732ed8a4-6885-479f-b579-4359355cc8ea"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:59:31.462517 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.462463    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732ed8a4-6885-479f-b579-4359355cc8ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "732ed8a4-6885-479f-b579-4359355cc8ea" (UID: "732ed8a4-6885-479f-b579-4359355cc8ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:59:31.462624 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.462607    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732ed8a4-6885-479f-b579-4359355cc8ea-kube-api-access-5zdvq" (OuterVolumeSpecName: "kube-api-access-5zdvq") pod "732ed8a4-6885-479f-b579-4359355cc8ea" (UID: "732ed8a4-6885-479f-b579-4359355cc8ea"). InnerVolumeSpecName "kube-api-access-5zdvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:59:31.470290 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.470265    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732ed8a4-6885-479f-b579-4359355cc8ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "732ed8a4-6885-479f-b579-4359355cc8ea" (UID: "732ed8a4-6885-479f-b579-4359355cc8ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:59:31.518352 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.518324    2563 generic.go:358] "Generic (PLEG): container finished" podID="732ed8a4-6885-479f-b579-4359355cc8ea" containerID="c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a" exitCode=0
May 26 11:59:31.518439 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.518402    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"
May 26 11:59:31.518439 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.518409    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" event={"ID":"732ed8a4-6885-479f-b579-4359355cc8ea","Type":"ContainerDied","Data":"c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a"}
May 26 11:59:31.518519 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.518449    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr" event={"ID":"732ed8a4-6885-479f-b579-4359355cc8ea","Type":"ContainerDied","Data":"b2b8d403e938aa6e3f7b534fce0745e27a4c581985044b522f90476b231cb020"}
May 26 11:59:31.518519 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.518464    2563 scope.go:117] "RemoveContainer" containerID="43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707"
May 26 11:59:31.526049 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.526028    2563 scope.go:117] "RemoveContainer" containerID="c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a"
May 26 11:59:31.532982 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.532966    2563 scope.go:117] "RemoveContainer" containerID="9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883"
May 26 11:59:31.540009 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.539987    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"]
May 26 11:59:31.540259 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.540240    2563 scope.go:117] "RemoveContainer" containerID="43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707"
May 26 11:59:31.540479 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:59:31.540463    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707\": container with ID starting with 43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707 not found: ID does not exist" containerID="43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707"
May 26 11:59:31.540525 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.540486    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707"} err="failed to get container status \"43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707\": rpc error: code = NotFound desc = could not find container \"43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707\": container with ID starting with 43a275ddac195a3b9a1d09e1111ca0a75d24b73f9566c4ee724bc5d3331ea707 not found: ID does not exist"
May 26 11:59:31.540525 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.540502    2563 scope.go:117] "RemoveContainer" containerID="c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a"
May 26 11:59:31.540729 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:59:31.540712    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a\": container with ID starting with c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a not found: ID does not exist" containerID="c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a"
May 26 11:59:31.540775 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.540734    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a"} err="failed to get container status \"c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a\": rpc error: code = NotFound desc = could not find container \"c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a\": container with ID starting with c688d83cb7bb15eaa570d86f6803bbaa667c39346c16788434af99519006c04a not found: ID does not exist"
May 26 11:59:31.540775 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.540750    2563 scope.go:117] "RemoveContainer" containerID="9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883"
May 26 11:59:31.540971 ip-10-0-138-207 kubenswrapper[2563]: E0526 11:59:31.540957    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883\": container with ID starting with 9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883 not found: ID does not exist" containerID="9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883"
May 26 11:59:31.541011 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.540973    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883"} err="failed to get container status \"9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883\": rpc error: code = NotFound desc = could not find container \"9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883\": container with ID starting with 9bfb690af6ff3b465aed69d66eb2f076c11b3cc159dfe1c2b8fd532ba494e883 not found: ID does not exist"
May 26 11:59:31.543057 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.543039    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-hdjbr"]
May 26 11:59:31.561573 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.561552    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zdvq\" (UniqueName: \"kubernetes.io/projected/732ed8a4-6885-479f-b579-4359355cc8ea-kube-api-access-5zdvq\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:59:31.561661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.561577    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732ed8a4-6885-479f-b579-4359355cc8ea-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:59:31.561661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.561593    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732ed8a4-6885-479f-b579-4359355cc8ea-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:59:31.561661 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:31.561608    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732ed8a4-6885-479f-b579-4359355cc8ea-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 11:59:32.242513 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:32.242483    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" path="/var/lib/kubelet/pods/732ed8a4-6885-479f-b579-4359355cc8ea/volumes"
May 26 11:59:33.526530 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:33.526496    2563 generic.go:358] "Generic (PLEG): container finished" podID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerID="b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1" exitCode=0
May 26 11:59:33.526897 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:33.526571    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" event={"ID":"5fe57b3b-b3bf-44b5-94fd-69082450e246","Type":"ContainerDied","Data":"b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1"}
May 26 11:59:40.549808 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:40.549767    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" event={"ID":"5fe57b3b-b3bf-44b5-94fd-69082450e246","Type":"ContainerStarted","Data":"64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0"}
May 26 11:59:41.555006 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:41.554969    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" event={"ID":"5fe57b3b-b3bf-44b5-94fd-69082450e246","Type":"ContainerStarted","Data":"731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72"}
May 26 11:59:41.555412 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:41.555183    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:41.555412 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:41.555216    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:41.556507 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:41.556480    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:59:41.572963 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:41.572923    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podStartSLOduration=6.648245252 podStartE2EDuration="13.572911232s" podCreationTimestamp="2026-05-26 11:59:28 +0000 UTC" firstStartedPulling="2026-05-26 11:59:33.527660034 +0000 UTC m=+1935.890609681" lastFinishedPulling="2026-05-26 11:59:40.452326013 +0000 UTC m=+1942.815275661" observedRunningTime="2026-05-26 11:59:41.571415982 +0000 UTC m=+1943.934365651" watchObservedRunningTime="2026-05-26 11:59:41.572911232 +0000 UTC m=+1943.935860932"
May 26 11:59:42.557856 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:42.557822    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:59:47.561781 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:47.561749    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 11:59:47.562353 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:47.562327    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:59:57.562452 ip-10-0-138-207 kubenswrapper[2563]: I0526 11:59:57.562411    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 12:00:07.563162 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:00:07.563074    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 12:00:17.562884 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:00:17.562838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 12:00:27.562809 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:00:27.562753    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 12:00:37.562517 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:00:37.562475    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 12:00:47.562938 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:00:47.562896    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 12:00:52.238543 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:00:52.238503    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 12:01:02.241691 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:02.241663    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 12:01:09.888029 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.887997    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"]
May 26 12:01:09.888434 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.888296    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" containerID="cri-o://64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0" gracePeriod=30
May 26 12:01:09.888434 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.888336    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kube-rbac-proxy" containerID="cri-o://731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72" gracePeriod=30
May 26 12:01:09.998555 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.998523    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"]
May 26 12:01:09.998884 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.998868    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container"
May 26 12:01:09.998987 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.998886    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container"
May 26 12:01:09.998987 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.998914    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="storage-initializer"
May 26 12:01:09.998987 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.998923    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="storage-initializer"
May 26 12:01:09.998987 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.998940    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kube-rbac-proxy"
May 26 12:01:09.998987 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.998948    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kube-rbac-proxy"
May 26 12:01:09.999239 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.999012    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kserve-container"
May 26 12:01:09.999239 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:09.999027    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="732ed8a4-6885-479f-b579-4359355cc8ea" containerName="kube-rbac-proxy"
May 26 12:01:10.002020 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.001999    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.003846 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.003825    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
May 26 12:01:10.003959 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.003931    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
May 26 12:01:10.012548 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.012524    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"]
May 26 12:01:10.089089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.089058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.089284 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.089095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.089284 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.089117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.089284 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.089197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjsj\" (UniqueName: \"kubernetes.io/projected/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kube-api-access-nfjsj\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.190061 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.189975    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.190061 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.190013    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.190061 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.190039    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.190340 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.190089    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjsj\" (UniqueName: \"kubernetes.io/projected/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kube-api-access-nfjsj\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.190396 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.190373    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.190681 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.190658    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.192621 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.192603    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.198828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.198781    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjsj\" (UniqueName: \"kubernetes.io/projected/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kube-api-access-nfjsj\") pod \"isvc-pmml-runtime-predictor-67bc544947-ddxlj\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.312566 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.312517    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:10.441731 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.441652    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"]
May 26 12:01:10.445009 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:01:10.444983    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec802df_e45f_4577_bcb9_8e5b88ca6fdf.slice/crio-886b884fcda688a6326a8fd810b34e9487eaeafe8fe4d993c6b14454d38e5c41 WatchSource:0}: Error finding container 886b884fcda688a6326a8fd810b34e9487eaeafe8fe4d993c6b14454d38e5c41: Status 404 returned error can't find the container with id 886b884fcda688a6326a8fd810b34e9487eaeafe8fe4d993c6b14454d38e5c41
May 26 12:01:10.789553 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.789478    2563 generic.go:358] "Generic (PLEG): container finished" podID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerID="731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72" exitCode=2
May 26 12:01:10.789700 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.789559    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" event={"ID":"5fe57b3b-b3bf-44b5-94fd-69082450e246","Type":"ContainerDied","Data":"731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72"}
May 26 12:01:10.790851 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.790821    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" event={"ID":"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf","Type":"ContainerStarted","Data":"84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48"}
May 26 12:01:10.790965 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:10.790861    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" event={"ID":"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf","Type":"ContainerStarted","Data":"886b884fcda688a6326a8fd810b34e9487eaeafe8fe4d993c6b14454d38e5c41"}
May 26 12:01:12.238916 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:12.238873    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 12:01:12.558134 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:12.558096    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
May 26 12:01:13.426878 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.426855    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 12:01:13.517265 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.517181    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe57b3b-b3bf-44b5-94fd-69082450e246-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"5fe57b3b-b3bf-44b5-94fd-69082450e246\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") "
May 26 12:01:13.517265 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.517243    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l8xp\" (UniqueName: \"kubernetes.io/projected/5fe57b3b-b3bf-44b5-94fd-69082450e246-kube-api-access-5l8xp\") pod \"5fe57b3b-b3bf-44b5-94fd-69082450e246\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") "
May 26 12:01:13.517458 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.517284    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe57b3b-b3bf-44b5-94fd-69082450e246-kserve-provision-location\") pod \"5fe57b3b-b3bf-44b5-94fd-69082450e246\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") "
May 26 12:01:13.517458 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.517330    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe57b3b-b3bf-44b5-94fd-69082450e246-proxy-tls\") pod \"5fe57b3b-b3bf-44b5-94fd-69082450e246\" (UID: \"5fe57b3b-b3bf-44b5-94fd-69082450e246\") "
May 26 12:01:13.517606 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.517586    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe57b3b-b3bf-44b5-94fd-69082450e246-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "5fe57b3b-b3bf-44b5-94fd-69082450e246" (UID: "5fe57b3b-b3bf-44b5-94fd-69082450e246"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:01:13.517675 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.517582    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe57b3b-b3bf-44b5-94fd-69082450e246-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5fe57b3b-b3bf-44b5-94fd-69082450e246" (UID: "5fe57b3b-b3bf-44b5-94fd-69082450e246"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:01:13.519414 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.519381    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe57b3b-b3bf-44b5-94fd-69082450e246-kube-api-access-5l8xp" (OuterVolumeSpecName: "kube-api-access-5l8xp") pod "5fe57b3b-b3bf-44b5-94fd-69082450e246" (UID: "5fe57b3b-b3bf-44b5-94fd-69082450e246"). InnerVolumeSpecName "kube-api-access-5l8xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:01:13.519518 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.519465    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe57b3b-b3bf-44b5-94fd-69082450e246-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fe57b3b-b3bf-44b5-94fd-69082450e246" (UID: "5fe57b3b-b3bf-44b5-94fd-69082450e246"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:01:13.618608 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.618580    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe57b3b-b3bf-44b5-94fd-69082450e246-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:01:13.618608 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.618610    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe57b3b-b3bf-44b5-94fd-69082450e246-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:01:13.618608 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.618626    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe57b3b-b3bf-44b5-94fd-69082450e246-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:01:13.618844 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.618639    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l8xp\" (UniqueName: \"kubernetes.io/projected/5fe57b3b-b3bf-44b5-94fd-69082450e246-kube-api-access-5l8xp\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:01:13.799661 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.799633    2563 generic.go:358] "Generic (PLEG): container finished" podID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerID="64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0" exitCode=0
May 26 12:01:13.799854 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.799717    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"
May 26 12:01:13.799854 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.799719    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" event={"ID":"5fe57b3b-b3bf-44b5-94fd-69082450e246","Type":"ContainerDied","Data":"64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0"}
May 26 12:01:13.799854 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.799763    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f" event={"ID":"5fe57b3b-b3bf-44b5-94fd-69082450e246","Type":"ContainerDied","Data":"a16494a972e8d3dbf0463e33cd31890041f6d138dd7212cfa7a178d9f4514a19"}
May 26 12:01:13.799854 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.799783    2563 scope.go:117] "RemoveContainer" containerID="731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72"
May 26 12:01:13.807409 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.807338    2563 scope.go:117] "RemoveContainer" containerID="64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0"
May 26 12:01:13.818605 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.818586    2563 scope.go:117] "RemoveContainer" containerID="b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1"
May 26 12:01:13.825459 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.825440    2563 scope.go:117] "RemoveContainer" containerID="731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72"
May 26 12:01:13.825644 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.825621    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"]
May 26 12:01:13.825740 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:01:13.825682    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72\": container with ID starting with 731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72 not found: ID does not exist" containerID="731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72"
May 26 12:01:13.825740 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.825708    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72"} err="failed to get container status \"731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72\": rpc error: code = NotFound desc = could not find container \"731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72\": container with ID starting with 731827774c308dbd7e4b05fedd7cff2fccc472d349aa31f52995ea0516e93f72 not found: ID does not exist"
May 26 12:01:13.825740 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.825724    2563 scope.go:117] "RemoveContainer" containerID="64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0"
May 26 12:01:13.825991 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:01:13.825970    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0\": container with ID starting with 64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0 not found: ID does not exist" containerID="64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0"
May 26 12:01:13.826034 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.825998    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0"} err="failed to get container status \"64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0\": rpc error: code = NotFound desc = could not find container \"64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0\": container with ID starting with 64a8f25d30c91cfb0d7cbf5ccddedcdd8367d94ec68103879a805091ff4451f0 not found: ID does not exist"
May 26 12:01:13.826034 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.826015    2563 scope.go:117] "RemoveContainer" containerID="b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1"
May 26 12:01:13.826269 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:01:13.826253    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1\": container with ID starting with b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1 not found: ID does not exist" containerID="b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1"
May 26 12:01:13.826328 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.826273    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1"} err="failed to get container status \"b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1\": rpc error: code = NotFound desc = could not find container \"b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1\": container with ID starting with b73bdb29977511acd87a633135d80850f239601407e1fa4ba5f5d9f54b82ffd1 not found: ID does not exist"
May 26 12:01:13.833104 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:13.833078    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-ldf5f"]
May 26 12:01:14.242423 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:14.242397    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" path="/var/lib/kubelet/pods/5fe57b3b-b3bf-44b5-94fd-69082450e246/volumes"
May 26 12:01:14.803121 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:14.803090    2563 generic.go:358] "Generic (PLEG): container finished" podID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerID="84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48" exitCode=0
May 26 12:01:14.803538 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:14.803164    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" event={"ID":"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf","Type":"ContainerDied","Data":"84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48"}
May 26 12:01:15.809196 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:15.809163    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" event={"ID":"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf","Type":"ContainerStarted","Data":"3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4"}
May 26 12:01:15.809581 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:15.809204    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" event={"ID":"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf","Type":"ContainerStarted","Data":"acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0"}
May 26 12:01:15.809581 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:15.809480    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:15.809678 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:15.809625    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:15.810786 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:15.810757    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:01:15.829852 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:15.829812    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podStartSLOduration=6.829783845 podStartE2EDuration="6.829783845s" podCreationTimestamp="2026-05-26 12:01:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:01:15.827784588 +0000 UTC m=+2038.190734257" watchObservedRunningTime="2026-05-26 12:01:15.829783845 +0000 UTC m=+2038.192733515"
May 26 12:01:16.812723 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:16.812676    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:01:17.815659 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:17.815619    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:01:22.820963 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:22.820930    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:01:22.821442 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:22.821417    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:01:32.822182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:32.822143    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:01:42.821533 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:42.821448    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:01:52.821594 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:01:52.821546    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:02:02.821518 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:02.821476    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:02:12.822077 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:12.822033    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:02:22.821605 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:22.821568    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:02:32.822941 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:32.822906    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:02:41.071619 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.071561    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"]
May 26 12:02:41.072223 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.071972    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" containerID="cri-o://acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0" gracePeriod=30
May 26 12:02:41.072223 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.072048    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kube-rbac-proxy" containerID="cri-o://3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4" gracePeriod=30
May 26 12:02:41.200581 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200550    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"]
May 26 12:02:41.200844 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200819    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container"
May 26 12:02:41.200844 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200845    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container"
May 26 12:02:41.200944 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kube-rbac-proxy"
May 26 12:02:41.200944 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200865    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kube-rbac-proxy"
May 26 12:02:41.200944 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="storage-initializer"
May 26 12:02:41.200944 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200884    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="storage-initializer"
May 26 12:02:41.200944 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200920    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kserve-container"
May 26 12:02:41.200944 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.200927    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="5fe57b3b-b3bf-44b5-94fd-69082450e246" containerName="kube-rbac-proxy"
May 26 12:02:41.203684 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.203666    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.205814 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.205774    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
May 26 12:02:41.205965 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.205947    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
May 26 12:02:41.215780 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.215756    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"]
May 26 12:02:41.263862 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.263822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.263862 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.263859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpsjt\" (UniqueName: \"kubernetes.io/projected/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kube-api-access-lpsjt\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.264057 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.263916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.264057 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.263996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.365142 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.365038    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.365142 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.365116    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.365379 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.365162    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.365379 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.365289    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lpsjt\" (UniqueName: \"kubernetes.io/projected/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kube-api-access-lpsjt\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.365501 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.365467    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.365782 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.365757    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.367760 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.367742    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.373913 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.373890    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpsjt\" (UniqueName: \"kubernetes.io/projected/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kube-api-access-lpsjt\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.513138 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.513097    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:41.647028 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:41.646951    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"]
May 26 12:02:41.650099 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:02:41.650073    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d68e6f8_fb21_48e4_bda0_7ac24fa1fc46.slice/crio-c6d6bcb1436bb947ac1811bab29a2a513afab23a25e140f0c4af91871a872659 WatchSource:0}: Error finding container c6d6bcb1436bb947ac1811bab29a2a513afab23a25e140f0c4af91871a872659: Status 404 returned error can't find the container with id c6d6bcb1436bb947ac1811bab29a2a513afab23a25e140f0c4af91871a872659
May 26 12:02:42.037287 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:42.037193    2563 generic.go:358] "Generic (PLEG): container finished" podID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerID="3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4" exitCode=2
May 26 12:02:42.037287 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:42.037275    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" event={"ID":"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf","Type":"ContainerDied","Data":"3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4"}
May 26 12:02:42.038545 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:42.038520    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" event={"ID":"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46","Type":"ContainerStarted","Data":"db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b"}
May 26 12:02:42.038545 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:42.038549    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" event={"ID":"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46","Type":"ContainerStarted","Data":"c6d6bcb1436bb947ac1811bab29a2a513afab23a25e140f0c4af91871a872659"}
May 26 12:02:42.816701 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:42.816659    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
May 26 12:02:42.822022 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:42.821996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 12:02:44.520171 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.520145    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:02:44.591445 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.591364    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-proxy-tls\") pod \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") "
May 26 12:02:44.591445 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.591417    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kserve-provision-location\") pod \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") "
May 26 12:02:44.591624 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.591449    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") "
May 26 12:02:44.591624 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.591550    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfjsj\" (UniqueName: \"kubernetes.io/projected/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kube-api-access-nfjsj\") pod \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\" (UID: \"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf\") "
May 26 12:02:44.591757 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.591724    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" (UID: "5ec802df-e45f-4577-bcb9-8e5b88ca6fdf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:02:44.591838 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.591777    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" (UID: "5ec802df-e45f-4577-bcb9-8e5b88ca6fdf"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:02:44.593678 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.593659    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" (UID: "5ec802df-e45f-4577-bcb9-8e5b88ca6fdf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:02:44.593729 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.593711    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kube-api-access-nfjsj" (OuterVolumeSpecName: "kube-api-access-nfjsj") pod "5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" (UID: "5ec802df-e45f-4577-bcb9-8e5b88ca6fdf"). InnerVolumeSpecName "kube-api-access-nfjsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:02:44.692658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.692629    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:02:44.692658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.692655    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:02:44.692881 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.692666    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjsj\" (UniqueName: \"kubernetes.io/projected/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-kube-api-access-nfjsj\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:02:44.692881 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:44.692676    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:02:45.050462 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.050426    2563 generic.go:358] "Generic (PLEG): container finished" podID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerID="acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0" exitCode=0
May 26 12:02:45.050627 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.050487    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" event={"ID":"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf","Type":"ContainerDied","Data":"acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0"}
May 26 12:02:45.050627 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.050509    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"
May 26 12:02:45.050627 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.050515    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj" event={"ID":"5ec802df-e45f-4577-bcb9-8e5b88ca6fdf","Type":"ContainerDied","Data":"886b884fcda688a6326a8fd810b34e9487eaeafe8fe4d993c6b14454d38e5c41"}
May 26 12:02:45.050627 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.050531    2563 scope.go:117] "RemoveContainer" containerID="3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4"
May 26 12:02:45.058052 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.057946    2563 scope.go:117] "RemoveContainer" containerID="acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0"
May 26 12:02:45.064740 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.064724    2563 scope.go:117] "RemoveContainer" containerID="84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48"
May 26 12:02:45.071059 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.071042    2563 scope.go:117] "RemoveContainer" containerID="3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4"
May 26 12:02:45.071293 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:02:45.071276    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4\": container with ID starting with 3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4 not found: ID does not exist" containerID="3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4"
May 26 12:02:45.071337 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.071300    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4"} err="failed to get container status \"3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4\": rpc error: code = NotFound desc = could not find container \"3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4\": container with ID starting with 3045ba9a125dcbc6613f361b186b3db495b2ed83ffa6bad1ebed2f0377a261b4 not found: ID does not exist"
May 26 12:02:45.071337 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.071317    2563 scope.go:117] "RemoveContainer" containerID="acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0"
May 26 12:02:45.071558 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:02:45.071541    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0\": container with ID starting with acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0 not found: ID does not exist" containerID="acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0"
May 26 12:02:45.071609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.071563    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0"} err="failed to get container status \"acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0\": rpc error: code = NotFound desc = could not find container \"acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0\": container with ID starting with acef67b7ea3d6240ff2081656a5c686e92d98b32fe366c7bc16bdb03d0820be0 not found: ID does not exist"
May 26 12:02:45.071609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.071579    2563 scope.go:117] "RemoveContainer" containerID="84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48"
May 26 12:02:45.071770 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:02:45.071755    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48\": container with ID starting with 84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48 not found: ID does not exist" containerID="84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48"
May 26 12:02:45.071832 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.071774    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48"} err="failed to get container status \"84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48\": rpc error: code = NotFound desc = could not find container \"84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48\": container with ID starting with 84c7d902a9a9ee3876f3a5c26545a034f6622818f8d98dc4ec7865c4ad613f48 not found: ID does not exist"
May 26 12:02:45.084591 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.084571    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"]
May 26 12:02:45.097853 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:45.097832    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ddxlj"]
May 26 12:02:46.055518 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:46.055489    2563 generic.go:358] "Generic (PLEG): container finished" podID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerID="db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b" exitCode=0
May 26 12:02:46.055898 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:46.055548    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" event={"ID":"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46","Type":"ContainerDied","Data":"db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b"}
May 26 12:02:46.242138 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:46.242106    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" path="/var/lib/kubelet/pods/5ec802df-e45f-4577-bcb9-8e5b88ca6fdf/volumes"
May 26 12:02:47.059842 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:47.059784    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" event={"ID":"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46","Type":"ContainerStarted","Data":"c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744"}
May 26 12:02:47.059842 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:47.059849    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" event={"ID":"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46","Type":"ContainerStarted","Data":"c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67"}
May 26 12:02:47.060325 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:47.060161    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:47.060325 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:47.060270    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:47.061446 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:47.061424    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:02:47.084386 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:47.084342    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podStartSLOduration=6.084331131 podStartE2EDuration="6.084331131s" podCreationTimestamp="2026-05-26 12:02:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:02:47.082709832 +0000 UTC m=+2129.445659501" watchObservedRunningTime="2026-05-26 12:02:47.084331131 +0000 UTC m=+2129.447280799"
May 26 12:02:48.063096 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:48.063054    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:02:53.067495 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:53.067464    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:02:53.067966 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:02:53.067930    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:03:01.383406 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:03:01.383377    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:03:01.385858 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:03:01.385782    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:03:03.068349 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:03:03.068308    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:03:13.067992 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:03:13.067943    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:03:23.068482 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:03:23.068438    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:03:33.068306 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:03:33.068268    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:03:43.068847 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:03:43.068785    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:03:53.068025 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:03:53.067984    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:04:03.069002 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:03.068968    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:04:12.204107 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.204072    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"]
May 26 12:04:12.204623 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.204397    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" containerID="cri-o://c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67" gracePeriod=30
May 26 12:04:12.204623 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.204432    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kube-rbac-proxy" containerID="cri-o://c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744" gracePeriod=30
May 26 12:04:12.357932 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.357900    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"]
May 26 12:04:12.358154 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.358143    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kube-rbac-proxy"
May 26 12:04:12.358197 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.358156    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kube-rbac-proxy"
May 26 12:04:12.358197 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.358164    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="storage-initializer"
May 26 12:04:12.358197 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.358169    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="storage-initializer"
May 26 12:04:12.358197 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.358187    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container"
May 26 12:04:12.358197 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.358193    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container"
May 26 12:04:12.358352 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.358230    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kube-rbac-proxy"
May 26 12:04:12.358352 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.358238    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="5ec802df-e45f-4577-bcb9-8e5b88ca6fdf" containerName="kserve-container"
May 26 12:04:12.361028 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.361012    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.363196 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.363166    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ba5923-predictor-serving-cert\""
May 26 12:04:12.363324 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.363270    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ba5923-kube-rbac-proxy-sar-config\""
May 26 12:04:12.370287 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.370265    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"]
May 26 12:04:12.439264 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.439228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lzd\" (UniqueName: \"kubernetes.io/projected/822209a3-bdb6-4c3f-b109-b56a23234103-kube-api-access-d9lzd\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.439448 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.439271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/822209a3-bdb6-4c3f-b109-b56a23234103-kserve-provision-location\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.439448 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.439301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822209a3-bdb6-4c3f-b109-b56a23234103-isvc-primary-ba5923-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.439448 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.439332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822209a3-bdb6-4c3f-b109-b56a23234103-proxy-tls\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.540441 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.540412    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lzd\" (UniqueName: \"kubernetes.io/projected/822209a3-bdb6-4c3f-b109-b56a23234103-kube-api-access-d9lzd\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.540617 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.540455    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/822209a3-bdb6-4c3f-b109-b56a23234103-kserve-provision-location\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.540617 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.540484    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822209a3-bdb6-4c3f-b109-b56a23234103-isvc-primary-ba5923-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.540617 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.540515    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822209a3-bdb6-4c3f-b109-b56a23234103-proxy-tls\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.540940 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.540920    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/822209a3-bdb6-4c3f-b109-b56a23234103-kserve-provision-location\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.541181 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.541159    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822209a3-bdb6-4c3f-b109-b56a23234103-isvc-primary-ba5923-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.543106 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.543086    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822209a3-bdb6-4c3f-b109-b56a23234103-proxy-tls\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.551047 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.551024    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lzd\" (UniqueName: \"kubernetes.io/projected/822209a3-bdb6-4c3f-b109-b56a23234103-kube-api-access-d9lzd\") pod \"isvc-primary-ba5923-predictor-86f446f48-tmk96\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") " pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.672058 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.672021    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:12.801968 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.801910    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"]
May 26 12:04:12.804402 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:04:12.804375    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822209a3_bdb6_4c3f_b109_b56a23234103.slice/crio-891d528b2684223d576bba8f2945e5716949c110949a8dcdd8a091a4e9538ac9 WatchSource:0}: Error finding container 891d528b2684223d576bba8f2945e5716949c110949a8dcdd8a091a4e9538ac9: Status 404 returned error can't find the container with id 891d528b2684223d576bba8f2945e5716949c110949a8dcdd8a091a4e9538ac9
May 26 12:04:12.806647 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:12.806631    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:04:13.063774 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:13.063681    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
May 26 12:04:13.068604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:13.068571    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 12:04:13.285259 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:13.285224    2563 generic.go:358] "Generic (PLEG): container finished" podID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerID="c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744" exitCode=2
May 26 12:04:13.285672 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:13.285293    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" event={"ID":"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46","Type":"ContainerDied","Data":"c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744"}
May 26 12:04:13.286655 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:13.286634    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" event={"ID":"822209a3-bdb6-4c3f-b109-b56a23234103","Type":"ContainerStarted","Data":"37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1"}
May 26 12:04:13.286720 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:13.286665    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" event={"ID":"822209a3-bdb6-4c3f-b109-b56a23234103","Type":"ContainerStarted","Data":"891d528b2684223d576bba8f2945e5716949c110949a8dcdd8a091a4e9538ac9"}
May 26 12:04:15.650709 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.650687    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:04:15.665163 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.665143    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kserve-provision-location\") pod \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") "
May 26 12:04:15.665249 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.665191    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-proxy-tls\") pod \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") "
May 26 12:04:15.665249 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.665243    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpsjt\" (UniqueName: \"kubernetes.io/projected/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kube-api-access-lpsjt\") pod \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") "
May 26 12:04:15.665352 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.665266    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\" (UID: \"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46\") "
May 26 12:04:15.665446 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.665424    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" (UID: "9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:04:15.665636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.665613    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" (UID: "9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:04:15.667316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.667292    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" (UID: "9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:04:15.667316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.667295    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kube-api-access-lpsjt" (OuterVolumeSpecName: "kube-api-access-lpsjt") pod "9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" (UID: "9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46"). InnerVolumeSpecName "kube-api-access-lpsjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:04:15.766175 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.766092    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:04:15.766175 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.766119    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpsjt\" (UniqueName: \"kubernetes.io/projected/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kube-api-access-lpsjt\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:04:15.766175 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.766129    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:04:15.766175 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:15.766139    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:04:16.296449 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.296416    2563 generic.go:358] "Generic (PLEG): container finished" podID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerID="c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67" exitCode=0
May 26 12:04:16.296604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.296499    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" event={"ID":"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46","Type":"ContainerDied","Data":"c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67"}
May 26 12:04:16.296604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.296520    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"
May 26 12:04:16.296604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.296539    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt" event={"ID":"9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46","Type":"ContainerDied","Data":"c6d6bcb1436bb947ac1811bab29a2a513afab23a25e140f0c4af91871a872659"}
May 26 12:04:16.296604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.296556    2563 scope.go:117] "RemoveContainer" containerID="c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744"
May 26 12:04:16.303887 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.303866    2563 scope.go:117] "RemoveContainer" containerID="c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67"
May 26 12:04:16.312902 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.312848    2563 scope.go:117] "RemoveContainer" containerID="db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b"
May 26 12:04:16.314442 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.314422    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"]
May 26 12:04:16.317382 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.317362    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rxztt"]
May 26 12:04:16.320373 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.320350    2563 scope.go:117] "RemoveContainer" containerID="c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744"
May 26 12:04:16.320596 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:04:16.320577    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744\": container with ID starting with c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744 not found: ID does not exist" containerID="c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744"
May 26 12:04:16.320668 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.320609    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744"} err="failed to get container status \"c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744\": rpc error: code = NotFound desc = could not find container \"c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744\": container with ID starting with c7248104fb05dec7d4ff4e5e22d68946afdbf4027b081e1d1cf096ba23c47744 not found: ID does not exist"
May 26 12:04:16.320668 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.320635    2563 scope.go:117] "RemoveContainer" containerID="c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67"
May 26 12:04:16.320902 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:04:16.320884    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67\": container with ID starting with c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67 not found: ID does not exist" containerID="c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67"
May 26 12:04:16.320948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.320909    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67"} err="failed to get container status \"c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67\": rpc error: code = NotFound desc = could not find container \"c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67\": container with ID starting with c231929388d333f6db38b11fd4713b9f397e84138f7198f2e3d6426ac6b3ce67 not found: ID does not exist"
May 26 12:04:16.320948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.320930    2563 scope.go:117] "RemoveContainer" containerID="db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b"
May 26 12:04:16.321136 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:04:16.321117    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b\": container with ID starting with db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b not found: ID does not exist" containerID="db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b"
May 26 12:04:16.321184 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:16.321143    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b"} err="failed to get container status \"db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b\": rpc error: code = NotFound desc = could not find container \"db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b\": container with ID starting with db367275f269294e6ae2b2966e2bdeaecf48bc2983542b5f49e56a737f11ed6b not found: ID does not exist"
May 26 12:04:17.301173 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:17.301139    2563 generic.go:358] "Generic (PLEG): container finished" podID="822209a3-bdb6-4c3f-b109-b56a23234103" containerID="37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1" exitCode=0
May 26 12:04:17.301556 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:17.301199    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" event={"ID":"822209a3-bdb6-4c3f-b109-b56a23234103","Type":"ContainerDied","Data":"37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1"}
May 26 12:04:18.241703 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:18.241671    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" path="/var/lib/kubelet/pods/9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46/volumes"
May 26 12:04:18.305364 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:18.305338    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" event={"ID":"822209a3-bdb6-4c3f-b109-b56a23234103","Type":"ContainerStarted","Data":"b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc"}
May 26 12:04:18.305723 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:18.305371    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" event={"ID":"822209a3-bdb6-4c3f-b109-b56a23234103","Type":"ContainerStarted","Data":"ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5"}
May 26 12:04:18.305723 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:18.305644    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:18.305849 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:18.305760    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:18.307132 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:18.307108    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 12:04:18.327022 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:18.326984    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podStartSLOduration=6.326971916 podStartE2EDuration="6.326971916s" podCreationTimestamp="2026-05-26 12:04:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:04:18.326070595 +0000 UTC m=+2220.689020264" watchObservedRunningTime="2026-05-26 12:04:18.326971916 +0000 UTC m=+2220.689921584"
May 26 12:04:19.308298 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:19.308255    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 12:04:24.312420 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:24.312388    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:04:24.312990 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:24.312965    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 12:04:34.313706 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:34.313670    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 12:04:44.313247 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:44.313166    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 12:04:54.312911 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:04:54.312872    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 12:05:04.313368 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:04.313314    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 12:05:14.313514 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:14.313474    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 12:05:24.313503 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:24.313471    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:05:32.479417 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479372    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"]
May 26 12:05:32.480048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479732    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="storage-initializer"
May 26 12:05:32.480048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479750    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="storage-initializer"
May 26 12:05:32.480048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479768    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container"
May 26 12:05:32.480048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479776    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container"
May 26 12:05:32.480048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479785    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kube-rbac-proxy"
May 26 12:05:32.480048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479812    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kube-rbac-proxy"
May 26 12:05:32.480048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479890    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kube-rbac-proxy"
May 26 12:05:32.480048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.479906    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d68e6f8-fb21-48e4-bda0-7ac24fa1fc46" containerName="kserve-container"
May 26 12:05:32.483081 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.483059    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.486060 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.486034    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ba5923-predictor-serving-cert\""
May 26 12:05:32.486209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.486095    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-ba5923-dockercfg-9qc7x\""
May 26 12:05:32.486209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.486154    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ba5923-kube-rbac-proxy-sar-config\""
May 26 12:05:32.486209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.486158    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-ba5923\""
May 26 12:05:32.486209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.486184    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 12:05:32.493336 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.493310    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"]
May 26 12:05:32.631037 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.631002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.631235 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.631043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdj6b\" (UniqueName: \"kubernetes.io/projected/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kube-api-access-jdj6b\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.631235 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.631065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kserve-provision-location\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.631235 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.631091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-isvc-secondary-ba5923-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.631235 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.631110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-cabundle-cert\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.731673 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.731581    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.731673 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.731619    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jdj6b\" (UniqueName: \"kubernetes.io/projected/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kube-api-access-jdj6b\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.731673 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.731666    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kserve-provision-location\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.731997 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.731706    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-isvc-secondary-ba5923-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.731997 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.731851    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-cabundle-cert\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.731997 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:05:32.731911    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-serving-cert: secret "isvc-secondary-ba5923-predictor-serving-cert" not found
May 26 12:05:32.732117 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:05:32.732022    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls podName:68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e nodeName:}" failed. No retries permitted until 2026-05-26 12:05:33.231991119 +0000 UTC m=+2295.594940771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls") pod "isvc-secondary-ba5923-predictor-7789865795-pbvwd" (UID: "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e") : secret "isvc-secondary-ba5923-predictor-serving-cert" not found
May 26 12:05:32.732161 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.732127    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kserve-provision-location\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.732421 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.732403    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-cabundle-cert\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.732458 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.732436    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-isvc-secondary-ba5923-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:32.742217 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:32.742191    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdj6b\" (UniqueName: \"kubernetes.io/projected/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kube-api-access-jdj6b\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:33.235674 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:33.235624    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:33.238394 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:33.238366    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls\") pod \"isvc-secondary-ba5923-predictor-7789865795-pbvwd\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") " pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:33.394039 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:33.393997    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:33.521888 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:33.521761    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"]
May 26 12:05:33.525899 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:05:33.525872    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dd8c3b_4cc1_4bac_8cdf_e140b5bc668e.slice/crio-b049b9b0af4868c966f4fe5d9700683dc67eefda8e56d4bf1a75ae5504385fb3 WatchSource:0}: Error finding container b049b9b0af4868c966f4fe5d9700683dc67eefda8e56d4bf1a75ae5504385fb3: Status 404 returned error can't find the container with id b049b9b0af4868c966f4fe5d9700683dc67eefda8e56d4bf1a75ae5504385fb3
May 26 12:05:34.500730 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:34.500690    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd" event={"ID":"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e","Type":"ContainerStarted","Data":"f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc"}
May 26 12:05:34.500730 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:34.500731    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd" event={"ID":"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e","Type":"ContainerStarted","Data":"b049b9b0af4868c966f4fe5d9700683dc67eefda8e56d4bf1a75ae5504385fb3"}
May 26 12:05:35.505087 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:35.505008    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba5923-predictor-7789865795-pbvwd_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e/storage-initializer/0.log"
May 26 12:05:35.505087 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:35.505044    2563 generic.go:358] "Generic (PLEG): container finished" podID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" containerID="f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc" exitCode=1
May 26 12:05:35.505479 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:35.505075    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd" event={"ID":"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e","Type":"ContainerDied","Data":"f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc"}
May 26 12:05:36.508920 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:36.508886    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba5923-predictor-7789865795-pbvwd_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e/storage-initializer/0.log"
May 26 12:05:36.509297 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:36.508956    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd" event={"ID":"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e","Type":"ContainerStarted","Data":"435b5265b11340ebdd9a54ac4a60e2ecbb78f5266df189f1561cc14eb0ff4bbc"}
May 26 12:05:39.517880 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:39.517851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba5923-predictor-7789865795-pbvwd_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e/storage-initializer/1.log"
May 26 12:05:39.518243 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:39.518206    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba5923-predictor-7789865795-pbvwd_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e/storage-initializer/0.log"
May 26 12:05:39.518243 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:39.518237    2563 generic.go:358] "Generic (PLEG): container finished" podID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" containerID="435b5265b11340ebdd9a54ac4a60e2ecbb78f5266df189f1561cc14eb0ff4bbc" exitCode=1
May 26 12:05:39.518324 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:39.518272    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd" event={"ID":"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e","Type":"ContainerDied","Data":"435b5265b11340ebdd9a54ac4a60e2ecbb78f5266df189f1561cc14eb0ff4bbc"}
May 26 12:05:39.518324 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:39.518300    2563 scope.go:117] "RemoveContainer" containerID="f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc"
May 26 12:05:39.518672 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:39.518653    2563 scope.go:117] "RemoveContainer" containerID="f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc"
May 26 12:05:39.528416 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:05:39.528385    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ba5923-predictor-7789865795-pbvwd_kserve-ci-e2e-test_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e_0 in pod sandbox b049b9b0af4868c966f4fe5d9700683dc67eefda8e56d4bf1a75ae5504385fb3 from index: no such id: 'f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc'" containerID="f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc"
May 26 12:05:39.528534 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:39.528427    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ba5923-predictor-7789865795-pbvwd_kserve-ci-e2e-test_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e_0 in pod sandbox b049b9b0af4868c966f4fe5d9700683dc67eefda8e56d4bf1a75ae5504385fb3 from index: no such id: 'f6f9230a1a255c87f769c322308de557528da5972191c5ada47f8b1f2ed22cfc'"
May 26 12:05:39.528594 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:05:39.528575    2563 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-ba5923-predictor-7789865795-pbvwd_kserve-ci-e2e-test(68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e)\"" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd" podUID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e"
May 26 12:05:40.522376 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:40.522345    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba5923-predictor-7789865795-pbvwd_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e/storage-initializer/1.log"
May 26 12:05:46.589279 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.589223    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"]
May 26 12:05:46.701920 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.701875    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"]
May 26 12:05:46.702325 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.702297    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container" containerID="cri-o://ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5" gracePeriod=30
May 26 12:05:46.702433 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.702333    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kube-rbac-proxy" containerID="cri-o://b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc" gracePeriod=30
May 26 12:05:46.744663 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.744630    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"]
May 26 12:05:46.748984 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.748964    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.750951 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.750931    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-cf24a5\""
May 26 12:05:46.751034 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.750931    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-cf24a5-dockercfg-2njph\""
May 26 12:05:46.751169 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.751153    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\""
May 26 12:05:46.751731 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.751708    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-cf24a5-predictor-serving-cert\""
May 26 12:05:46.761401 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.761376    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"]
May 26 12:05:46.820426 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.820407    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba5923-predictor-7789865795-pbvwd_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e/storage-initializer/1.log"
May 26 12:05:46.820533 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.820471    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:46.837987 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.837962    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-cabundle-cert\") pod \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") "
May 26 12:05:46.838105 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838005    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls\") pod \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") "
May 26 12:05:46.838105 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838051    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-isvc-secondary-ba5923-kube-rbac-proxy-sar-config\") pod \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") "
May 26 12:05:46.838105 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838075    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdj6b\" (UniqueName: \"kubernetes.io/projected/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kube-api-access-jdj6b\") pod \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") "
May 26 12:05:46.838105 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838102    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kserve-provision-location\") pod \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\" (UID: \"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e\") "
May 26 12:05:46.838316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kserve-provision-location\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.838316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876l8\" (UniqueName: \"kubernetes.io/projected/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kube-api-access-876l8\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.838316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.838472 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-proxy-tls\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.838472 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838394    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" (UID: "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:46.838576 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-cabundle-cert\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.838576 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838497    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-isvc-secondary-ba5923-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-ba5923-kube-rbac-proxy-sar-config") pod "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" (UID: "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e"). InnerVolumeSpecName "isvc-secondary-ba5923-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:46.838576 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838508    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" (UID: "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:05:46.838576 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.838537    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-cabundle-cert\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:46.840431 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.840380    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kube-api-access-jdj6b" (OuterVolumeSpecName: "kube-api-access-jdj6b") pod "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" (UID: "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e"). InnerVolumeSpecName "kube-api-access-jdj6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:05:46.840497 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.840473    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" (UID: "68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:05:46.939911 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.939868    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-proxy-tls\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.940102 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.939937    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-cabundle-cert\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.940102 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.939977    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kserve-provision-location\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.940102 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940002    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-876l8\" (UniqueName: \"kubernetes.io/projected/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kube-api-access-876l8\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.940102 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940033    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.940102 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940073    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-isvc-secondary-ba5923-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:46.940102 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940089    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdj6b\" (UniqueName: \"kubernetes.io/projected/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kube-api-access-jdj6b\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:46.940102 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940101    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:46.940501 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940110    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:46.940501 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940481    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kserve-provision-location\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.940706 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940685    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-cabundle-cert\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.940783 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.940766    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.942432 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.942415    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-proxy-tls\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:46.951327 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:46.951302    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-876l8\" (UniqueName: \"kubernetes.io/projected/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kube-api-access-876l8\") pod \"isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") " pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:47.059109 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.059059    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:47.189980 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.189946    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"]
May 26 12:05:47.193225 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:05:47.193202    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ad3e2d_3fea_49f9_88ca_a8138b9e17f9.slice/crio-2f61b694020d7c8d4ecdc77c14cf638010e73d41e7d8a11383ae9e8b76a51bf7 WatchSource:0}: Error finding container 2f61b694020d7c8d4ecdc77c14cf638010e73d41e7d8a11383ae9e8b76a51bf7: Status 404 returned error can't find the container with id 2f61b694020d7c8d4ecdc77c14cf638010e73d41e7d8a11383ae9e8b76a51bf7
May 26 12:05:47.541736 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.541701    2563 generic.go:358] "Generic (PLEG): container finished" podID="822209a3-bdb6-4c3f-b109-b56a23234103" containerID="b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc" exitCode=2
May 26 12:05:47.541913 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.541781    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" event={"ID":"822209a3-bdb6-4c3f-b109-b56a23234103","Type":"ContainerDied","Data":"b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc"}
May 26 12:05:47.542924 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.542905    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba5923-predictor-7789865795-pbvwd_68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e/storage-initializer/1.log"
May 26 12:05:47.543033 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.542972    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd" event={"ID":"68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e","Type":"ContainerDied","Data":"b049b9b0af4868c966f4fe5d9700683dc67eefda8e56d4bf1a75ae5504385fb3"}
May 26 12:05:47.543033 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.543002    2563 scope.go:117] "RemoveContainer" containerID="435b5265b11340ebdd9a54ac4a60e2ecbb78f5266df189f1561cc14eb0ff4bbc"
May 26 12:05:47.543153 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.543038    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"
May 26 12:05:47.544515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.544490    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z" event={"ID":"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9","Type":"ContainerStarted","Data":"0a7ee22c93ddf56e22cca18769bf194bd1ef118399eacbc25368775b634375c0"}
May 26 12:05:47.544608 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.544526    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z" event={"ID":"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9","Type":"ContainerStarted","Data":"2f61b694020d7c8d4ecdc77c14cf638010e73d41e7d8a11383ae9e8b76a51bf7"}
May 26 12:05:47.600570 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.600540    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"]
May 26 12:05:47.608783 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:47.608750    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba5923-predictor-7789865795-pbvwd"]
May 26 12:05:48.243041 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:48.242998    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" path="/var/lib/kubelet/pods/68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e/volumes"
May 26 12:05:49.308876 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:49.308832    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
May 26 12:05:50.557589 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:50.557558    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z_f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9/storage-initializer/0.log"
May 26 12:05:50.557990 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:50.557597    2563 generic.go:358] "Generic (PLEG): container finished" podID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" containerID="0a7ee22c93ddf56e22cca18769bf194bd1ef118399eacbc25368775b634375c0" exitCode=1
May 26 12:05:50.557990 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:50.557636    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z" event={"ID":"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9","Type":"ContainerDied","Data":"0a7ee22c93ddf56e22cca18769bf194bd1ef118399eacbc25368775b634375c0"}
May 26 12:05:51.158158 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.158134    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:05:51.275677 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.275579    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822209a3-bdb6-4c3f-b109-b56a23234103-proxy-tls\") pod \"822209a3-bdb6-4c3f-b109-b56a23234103\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") "
May 26 12:05:51.275677 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.275627    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822209a3-bdb6-4c3f-b109-b56a23234103-isvc-primary-ba5923-kube-rbac-proxy-sar-config\") pod \"822209a3-bdb6-4c3f-b109-b56a23234103\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") "
May 26 12:05:51.275677 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.275652    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/822209a3-bdb6-4c3f-b109-b56a23234103-kserve-provision-location\") pod \"822209a3-bdb6-4c3f-b109-b56a23234103\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") "
May 26 12:05:51.276074 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.275702    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9lzd\" (UniqueName: \"kubernetes.io/projected/822209a3-bdb6-4c3f-b109-b56a23234103-kube-api-access-d9lzd\") pod \"822209a3-bdb6-4c3f-b109-b56a23234103\" (UID: \"822209a3-bdb6-4c3f-b109-b56a23234103\") "
May 26 12:05:51.276133 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.276069    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822209a3-bdb6-4c3f-b109-b56a23234103-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "822209a3-bdb6-4c3f-b109-b56a23234103" (UID: "822209a3-bdb6-4c3f-b109-b56a23234103"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:05:51.276133 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.276086    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822209a3-bdb6-4c3f-b109-b56a23234103-isvc-primary-ba5923-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-ba5923-kube-rbac-proxy-sar-config") pod "822209a3-bdb6-4c3f-b109-b56a23234103" (UID: "822209a3-bdb6-4c3f-b109-b56a23234103"). InnerVolumeSpecName "isvc-primary-ba5923-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:51.277977 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.277944    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822209a3-bdb6-4c3f-b109-b56a23234103-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "822209a3-bdb6-4c3f-b109-b56a23234103" (UID: "822209a3-bdb6-4c3f-b109-b56a23234103"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:05:51.278048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.277991    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822209a3-bdb6-4c3f-b109-b56a23234103-kube-api-access-d9lzd" (OuterVolumeSpecName: "kube-api-access-d9lzd") pod "822209a3-bdb6-4c3f-b109-b56a23234103" (UID: "822209a3-bdb6-4c3f-b109-b56a23234103"). InnerVolumeSpecName "kube-api-access-d9lzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:05:51.376557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.376515    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822209a3-bdb6-4c3f-b109-b56a23234103-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:51.376557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.376551    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-ba5923-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822209a3-bdb6-4c3f-b109-b56a23234103-isvc-primary-ba5923-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:51.376557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.376562    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/822209a3-bdb6-4c3f-b109-b56a23234103-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:51.376837 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.376573    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9lzd\" (UniqueName: \"kubernetes.io/projected/822209a3-bdb6-4c3f-b109-b56a23234103-kube-api-access-d9lzd\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:51.562373 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.562347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z_f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9/storage-initializer/0.log"
May 26 12:05:51.562786 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.562462    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z" event={"ID":"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9","Type":"ContainerStarted","Data":"216a11582a5c44625f259feb821a981c980800659d7a51781e04a81ea6e36e58"}
May 26 12:05:51.564114 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.564088    2563 generic.go:358] "Generic (PLEG): container finished" podID="822209a3-bdb6-4c3f-b109-b56a23234103" containerID="ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5" exitCode=0
May 26 12:05:51.564201 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.564141    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" event={"ID":"822209a3-bdb6-4c3f-b109-b56a23234103","Type":"ContainerDied","Data":"ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5"}
May 26 12:05:51.564201 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.564153    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"
May 26 12:05:51.564201 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.564174    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96" event={"ID":"822209a3-bdb6-4c3f-b109-b56a23234103","Type":"ContainerDied","Data":"891d528b2684223d576bba8f2945e5716949c110949a8dcdd8a091a4e9538ac9"}
May 26 12:05:51.564201 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.564191    2563 scope.go:117] "RemoveContainer" containerID="b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc"
May 26 12:05:51.572909 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.572886    2563 scope.go:117] "RemoveContainer" containerID="ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5"
May 26 12:05:51.580171 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.580151    2563 scope.go:117] "RemoveContainer" containerID="37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1"
May 26 12:05:51.587662 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.587641    2563 scope.go:117] "RemoveContainer" containerID="b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc"
May 26 12:05:51.588284 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:05:51.588261    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc\": container with ID starting with b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc not found: ID does not exist" containerID="b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc"
May 26 12:05:51.588386 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.588295    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc"} err="failed to get container status \"b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc\": rpc error: code = NotFound desc = could not find container \"b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc\": container with ID starting with b7d4424c306704346a4241a4d848761b3510a5ca2b48d7164c68b584971865cc not found: ID does not exist"
May 26 12:05:51.588386 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.588321    2563 scope.go:117] "RemoveContainer" containerID="ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5"
May 26 12:05:51.588629 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:05:51.588604    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5\": container with ID starting with ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5 not found: ID does not exist" containerID="ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5"
May 26 12:05:51.588739 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.588640    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5"} err="failed to get container status \"ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5\": rpc error: code = NotFound desc = could not find container \"ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5\": container with ID starting with ddd72d0a719822e9b1392e494071aeb6102d95711a351b29b3d353d586fde3b5 not found: ID does not exist"
May 26 12:05:51.588739 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.588662    2563 scope.go:117] "RemoveContainer" containerID="37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1"
May 26 12:05:51.588985 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:05:51.588959    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1\": container with ID starting with 37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1 not found: ID does not exist" containerID="37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1"
May 26 12:05:51.589035 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.588992    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1"} err="failed to get container status \"37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1\": rpc error: code = NotFound desc = could not find container \"37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1\": container with ID starting with 37d4745ab685d5f2c6052981be57a13a4712a1dbfd680a03d3578e4055e5feb1 not found: ID does not exist"
May 26 12:05:51.605831 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.605781    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"]
May 26 12:05:51.616142 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.616111    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba5923-predictor-86f446f48-tmk96"]
May 26 12:05:51.755046 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.755009    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"]
May 26 12:05:51.871466 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871377    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"]
May 26 12:05:51.871751 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="storage-initializer"
May 26 12:05:51.871860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871754    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="storage-initializer"
May 26 12:05:51.871860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871774    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" containerName="storage-initializer"
May 26 12:05:51.871860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871783    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" containerName="storage-initializer"
May 26 12:05:51.871860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871814    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kube-rbac-proxy"
May 26 12:05:51.871860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871823    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kube-rbac-proxy"
May 26 12:05:51.871860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871841    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container"
May 26 12:05:51.871860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871849    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container"
May 26 12:05:51.872209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871918    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" containerName="storage-initializer"
May 26 12:05:51.872209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871934    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kube-rbac-proxy"
May 26 12:05:51.872209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.871944    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" containerName="kserve-container"
May 26 12:05:51.872209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.872010    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" containerName="storage-initializer"
May 26 12:05:51.872209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.872020    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" containerName="storage-initializer"
May 26 12:05:51.872209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.872110    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="68dd8c3b-4cc1-4bac-8cdf-e140b5bc668e" containerName="storage-initializer"
May 26 12:05:51.876385 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.876365    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:51.878756 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.878733    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
May 26 12:05:51.878874 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.878759    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
May 26 12:05:51.879169 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.879153    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b5sxz\""
May 26 12:05:51.886824 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.886780    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"]
May 26 12:05:51.980544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.980499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e212988e-e671-4e0c-acdc-a04a86197d96-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:51.980544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.980541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9jn\" (UniqueName: \"kubernetes.io/projected/e212988e-e671-4e0c-acdc-a04a86197d96-kube-api-access-7j9jn\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:51.980772 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.980582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e212988e-e671-4e0c-acdc-a04a86197d96-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:51.980772 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:51.980629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e212988e-e671-4e0c-acdc-a04a86197d96-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.081385 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.081337    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e212988e-e671-4e0c-acdc-a04a86197d96-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.081539 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.081452    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e212988e-e671-4e0c-acdc-a04a86197d96-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.081539 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.081486    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9jn\" (UniqueName: \"kubernetes.io/projected/e212988e-e671-4e0c-acdc-a04a86197d96-kube-api-access-7j9jn\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.081539 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.081529    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e212988e-e671-4e0c-acdc-a04a86197d96-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.081872 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.081847    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e212988e-e671-4e0c-acdc-a04a86197d96-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.082126 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.082109    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e212988e-e671-4e0c-acdc-a04a86197d96-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.084348 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.084325    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e212988e-e671-4e0c-acdc-a04a86197d96-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.090816 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.090765    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9jn\" (UniqueName: \"kubernetes.io/projected/e212988e-e671-4e0c-acdc-a04a86197d96-kube-api-access-7j9jn\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.186839 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.186780    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:05:52.244357 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.244314    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="822209a3-bdb6-4c3f-b109-b56a23234103" path="/var/lib/kubelet/pods/822209a3-bdb6-4c3f-b109-b56a23234103/volumes"
May 26 12:05:52.318181 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.318129    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"]
May 26 12:05:52.320807 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:05:52.320759    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode212988e_e671_4e0c_acdc_a04a86197d96.slice/crio-51cee9dac72b9bad4b26bf7e92bcafab9bbe6d607b7edbfebe3d9d843c8d409d WatchSource:0}: Error finding container 51cee9dac72b9bad4b26bf7e92bcafab9bbe6d607b7edbfebe3d9d843c8d409d: Status 404 returned error can't find the container with id 51cee9dac72b9bad4b26bf7e92bcafab9bbe6d607b7edbfebe3d9d843c8d409d
May 26 12:05:52.568940 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.568914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z_f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9/storage-initializer/1.log"
May 26 12:05:52.569338 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.569208    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z_f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9/storage-initializer/0.log"
May 26 12:05:52.569338 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.569241    2563 generic.go:358] "Generic (PLEG): container finished" podID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" containerID="216a11582a5c44625f259feb821a981c980800659d7a51781e04a81ea6e36e58" exitCode=1
May 26 12:05:52.569338 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.569325    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z" event={"ID":"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9","Type":"ContainerDied","Data":"216a11582a5c44625f259feb821a981c980800659d7a51781e04a81ea6e36e58"}
May 26 12:05:52.569502 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.569366    2563 scope.go:117] "RemoveContainer" containerID="0a7ee22c93ddf56e22cca18769bf194bd1ef118399eacbc25368775b634375c0"
May 26 12:05:52.571010 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.570986    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" event={"ID":"e212988e-e671-4e0c-acdc-a04a86197d96","Type":"ContainerStarted","Data":"0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134"}
May 26 12:05:52.571097 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.571020    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" event={"ID":"e212988e-e671-4e0c-acdc-a04a86197d96","Type":"ContainerStarted","Data":"51cee9dac72b9bad4b26bf7e92bcafab9bbe6d607b7edbfebe3d9d843c8d409d"}
May 26 12:05:52.696762 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.696738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z_f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9/storage-initializer/1.log"
May 26 12:05:52.696953 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.696817    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:52.789682 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.789634    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-proxy-tls\") pod \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") "
May 26 12:05:52.789682 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.789670    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-cabundle-cert\") pod \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") "
May 26 12:05:52.789969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.789708    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-876l8\" (UniqueName: \"kubernetes.io/projected/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kube-api-access-876l8\") pod \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") "
May 26 12:05:52.789969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.789751    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kserve-provision-location\") pod \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") "
May 26 12:05:52.789969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.789828    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\") pod \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\" (UID: \"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9\") "
May 26 12:05:52.790141 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.790112    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" (UID: "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:05:52.790203 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.790145    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" (UID: "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:52.790262 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.790220    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config") pod "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" (UID: "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9"). InnerVolumeSpecName "isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:05:52.792042 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.792012    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" (UID: "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:05:52.792463 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.792443    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kube-api-access-876l8" (OuterVolumeSpecName: "kube-api-access-876l8") pod "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" (UID: "f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9"). InnerVolumeSpecName "kube-api-access-876l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:05:52.890996 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.890908    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:52.890996 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.890938    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-cabundle-cert\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:52.890996 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.890948    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-876l8\" (UniqueName: \"kubernetes.io/projected/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kube-api-access-876l8\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:52.890996 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.890958    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:52.890996 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:52.890967    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9-isvc-init-fail-cf24a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:05:53.575984 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:53.575958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z_f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9/storage-initializer/1.log"
May 26 12:05:53.576367 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:53.576074    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"
May 26 12:05:53.576367 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:53.576100    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z" event={"ID":"f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9","Type":"ContainerDied","Data":"2f61b694020d7c8d4ecdc77c14cf638010e73d41e7d8a11383ae9e8b76a51bf7"}
May 26 12:05:53.576367 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:53.576139    2563 scope.go:117] "RemoveContainer" containerID="216a11582a5c44625f259feb821a981c980800659d7a51781e04a81ea6e36e58"
May 26 12:05:53.615344 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:53.615316    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"]
May 26 12:05:53.619739 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:53.619715    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-cf24a5-predictor-685f7d5b5f-srn4z"]
May 26 12:05:54.242507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:54.242472    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" path="/var/lib/kubelet/pods/f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9/volumes"
May 26 12:05:56.590316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:56.590284    2563 generic.go:358] "Generic (PLEG): container finished" podID="e212988e-e671-4e0c-acdc-a04a86197d96" containerID="0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134" exitCode=0
May 26 12:05:56.590738 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:05:56.590360    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" event={"ID":"e212988e-e671-4e0c-acdc-a04a86197d96","Type":"ContainerDied","Data":"0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134"}
May 26 12:06:21.672450 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:21.672413    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" event={"ID":"e212988e-e671-4e0c-acdc-a04a86197d96","Type":"ContainerStarted","Data":"64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913"}
May 26 12:06:21.672867 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:21.672462    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" event={"ID":"e212988e-e671-4e0c-acdc-a04a86197d96","Type":"ContainerStarted","Data":"e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce"}
May 26 12:06:21.672867 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:21.672676    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:06:21.695503 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:21.695450    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podStartSLOduration=6.246174912 podStartE2EDuration="30.695434094s" podCreationTimestamp="2026-05-26 12:05:51 +0000 UTC" firstStartedPulling="2026-05-26 12:05:56.591515421 +0000 UTC m=+2318.954465068" lastFinishedPulling="2026-05-26 12:06:21.040774598 +0000 UTC m=+2343.403724250" observedRunningTime="2026-05-26 12:06:21.692296067 +0000 UTC m=+2344.055245736" watchObservedRunningTime="2026-05-26 12:06:21.695434094 +0000 UTC m=+2344.058383765"
May 26 12:06:22.675479 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:22.675444    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:06:22.676594 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:22.676570    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:06:23.678219 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:23.678172    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:06:28.683859 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:28.683829    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:06:28.684446 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:28.684407    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:06:38.685020 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:38.684979    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:06:48.685012 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:48.684973    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:06:58.684393 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:06:58.684350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:07:08.685345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:08.685306    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:07:18.684897 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:18.684861    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:07:28.684850 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:28.684809    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 12:07:38.686045 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:38.685970    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:07:42.011619 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.011588    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"]
May 26 12:07:42.012141 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.012113    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container" containerID="cri-o://e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce" gracePeriod=30
May 26 12:07:42.012232 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.012183    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kube-rbac-proxy" containerID="cri-o://64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913" gracePeriod=30
May 26 12:07:42.137327 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.137295    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"]
May 26 12:07:42.137555 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.137542    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" containerName="storage-initializer"
May 26 12:07:42.137597 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.137556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" containerName="storage-initializer"
May 26 12:07:42.137597 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.137568    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" containerName="storage-initializer"
May 26 12:07:42.137597 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.137573    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" containerName="storage-initializer"
May 26 12:07:42.141703 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.138144    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" containerName="storage-initializer"
May 26 12:07:42.141703 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.138485    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9ad3e2d-3fea-49f9-88ca-a8138b9e17f9" containerName="storage-initializer"
May 26 12:07:42.142943 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.142922    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.144966 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.144936    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
May 26 12:07:42.145083 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.145058    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
May 26 12:07:42.153403 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.153382    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"]
May 26 12:07:42.188533 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.188504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.188672 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.188542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9867c6-0656-43f4-a6bc-ec8723f075d6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.188672 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.188573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9867c6-0656-43f4-a6bc-ec8723f075d6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.188672 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.188619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ch6b\" (UniqueName: \"kubernetes.io/projected/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kube-api-access-5ch6b\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.289917 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.289839    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.289917 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.289875    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9867c6-0656-43f4-a6bc-ec8723f075d6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.289917 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.289907    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9867c6-0656-43f4-a6bc-ec8723f075d6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.290211 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.289954    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5ch6b\" (UniqueName: \"kubernetes.io/projected/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kube-api-access-5ch6b\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.290307 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.290283    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.290553 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.290535    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9867c6-0656-43f4-a6bc-ec8723f075d6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.292616 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.292585    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9867c6-0656-43f4-a6bc-ec8723f075d6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.299936 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.299904    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ch6b\" (UniqueName: \"kubernetes.io/projected/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kube-api-access-5ch6b\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.452675 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.452644    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:42.582138 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.582105    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"]
May 26 12:07:42.585507 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:07:42.585481    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9867c6_0656_43f4_a6bc_ec8723f075d6.slice/crio-8cf40c8cc248b6b701ca4b8dcd423e81c977aee85f0a5cb433d4c882c02117fe WatchSource:0}: Error finding container 8cf40c8cc248b6b701ca4b8dcd423e81c977aee85f0a5cb433d4c882c02117fe: Status 404 returned error can't find the container with id 8cf40c8cc248b6b701ca4b8dcd423e81c977aee85f0a5cb433d4c882c02117fe
May 26 12:07:42.885386 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.885291    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" event={"ID":"ac9867c6-0656-43f4-a6bc-ec8723f075d6","Type":"ContainerStarted","Data":"3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc"}
May 26 12:07:42.885386 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.885336    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" event={"ID":"ac9867c6-0656-43f4-a6bc-ec8723f075d6","Type":"ContainerStarted","Data":"8cf40c8cc248b6b701ca4b8dcd423e81c977aee85f0a5cb433d4c882c02117fe"}
May 26 12:07:42.887298 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.887273    2563 generic.go:358] "Generic (PLEG): container finished" podID="e212988e-e671-4e0c-acdc-a04a86197d96" containerID="64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913" exitCode=2
May 26 12:07:42.887423 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:42.887349    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" event={"ID":"e212988e-e671-4e0c-acdc-a04a86197d96","Type":"ContainerDied","Data":"64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913"}
May 26 12:07:43.679036 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:43.678991    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
May 26 12:07:46.651777 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.651757    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:07:46.723820 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.723772    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j9jn\" (UniqueName: \"kubernetes.io/projected/e212988e-e671-4e0c-acdc-a04a86197d96-kube-api-access-7j9jn\") pod \"e212988e-e671-4e0c-acdc-a04a86197d96\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") "
May 26 12:07:46.723999 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.723848    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e212988e-e671-4e0c-acdc-a04a86197d96-kserve-provision-location\") pod \"e212988e-e671-4e0c-acdc-a04a86197d96\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") "
May 26 12:07:46.723999 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.723903    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e212988e-e671-4e0c-acdc-a04a86197d96-proxy-tls\") pod \"e212988e-e671-4e0c-acdc-a04a86197d96\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") "
May 26 12:07:46.723999 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.723946    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e212988e-e671-4e0c-acdc-a04a86197d96-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"e212988e-e671-4e0c-acdc-a04a86197d96\" (UID: \"e212988e-e671-4e0c-acdc-a04a86197d96\") "
May 26 12:07:46.724227 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.724194    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e212988e-e671-4e0c-acdc-a04a86197d96-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "e212988e-e671-4e0c-acdc-a04a86197d96" (UID: "e212988e-e671-4e0c-acdc-a04a86197d96"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:07:46.724227 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.724194    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e212988e-e671-4e0c-acdc-a04a86197d96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e212988e-e671-4e0c-acdc-a04a86197d96" (UID: "e212988e-e671-4e0c-acdc-a04a86197d96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:07:46.725989 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.725965    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e212988e-e671-4e0c-acdc-a04a86197d96-kube-api-access-7j9jn" (OuterVolumeSpecName: "kube-api-access-7j9jn") pod "e212988e-e671-4e0c-acdc-a04a86197d96" (UID: "e212988e-e671-4e0c-acdc-a04a86197d96"). InnerVolumeSpecName "kube-api-access-7j9jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:07:46.726057 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.726038    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e212988e-e671-4e0c-acdc-a04a86197d96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e212988e-e671-4e0c-acdc-a04a86197d96" (UID: "e212988e-e671-4e0c-acdc-a04a86197d96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:07:46.824994 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.824963    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e212988e-e671-4e0c-acdc-a04a86197d96-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:07:46.824994 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.824989    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e212988e-e671-4e0c-acdc-a04a86197d96-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:07:46.824994 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.825000    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j9jn\" (UniqueName: \"kubernetes.io/projected/e212988e-e671-4e0c-acdc-a04a86197d96-kube-api-access-7j9jn\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:07:46.825220 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.825009    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e212988e-e671-4e0c-acdc-a04a86197d96-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:07:46.899354 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.899284    2563 generic.go:358] "Generic (PLEG): container finished" podID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerID="3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc" exitCode=0
May 26 12:07:46.899474 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.899364    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" event={"ID":"ac9867c6-0656-43f4-a6bc-ec8723f075d6","Type":"ContainerDied","Data":"3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc"}
May 26 12:07:46.901052 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.901030    2563 generic.go:358] "Generic (PLEG): container finished" podID="e212988e-e671-4e0c-acdc-a04a86197d96" containerID="e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce" exitCode=0
May 26 12:07:46.901144 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.901056    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" event={"ID":"e212988e-e671-4e0c-acdc-a04a86197d96","Type":"ContainerDied","Data":"e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce"}
May 26 12:07:46.901144 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.901094    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp" event={"ID":"e212988e-e671-4e0c-acdc-a04a86197d96","Type":"ContainerDied","Data":"51cee9dac72b9bad4b26bf7e92bcafab9bbe6d607b7edbfebe3d9d843c8d409d"}
May 26 12:07:46.901144 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.901116    2563 scope.go:117] "RemoveContainer" containerID="64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913"
May 26 12:07:46.901144 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.901125    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"
May 26 12:07:46.908939 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.908911    2563 scope.go:117] "RemoveContainer" containerID="e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce"
May 26 12:07:46.915680 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.915658    2563 scope.go:117] "RemoveContainer" containerID="0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134"
May 26 12:07:46.925070 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.925046    2563 scope.go:117] "RemoveContainer" containerID="64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913"
May 26 12:07:46.925371 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:07:46.925339    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913\": container with ID starting with 64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913 not found: ID does not exist" containerID="64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913"
May 26 12:07:46.925455 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.925371    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913"} err="failed to get container status \"64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913\": rpc error: code = NotFound desc = could not find container \"64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913\": container with ID starting with 64ba123e35abe4df5282161f5097cbf8e5012129a29864f147b2ba894fdb2913 not found: ID does not exist"
May 26 12:07:46.925455 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.925389    2563 scope.go:117] "RemoveContainer" containerID="e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce"
May 26 12:07:46.925692 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:07:46.925666    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce\": container with ID starting with e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce not found: ID does not exist" containerID="e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce"
May 26 12:07:46.925764 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.925703    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce"} err="failed to get container status \"e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce\": rpc error: code = NotFound desc = could not find container \"e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce\": container with ID starting with e0bcca10d2d76d15a499847103529138c51f7f6e93a2503c40a6f21e2b3d23ce not found: ID does not exist"
May 26 12:07:46.925764 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.925726    2563 scope.go:117] "RemoveContainer" containerID="0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134"
May 26 12:07:46.927845 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:07:46.926081    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134\": container with ID starting with 0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134 not found: ID does not exist" containerID="0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134"
May 26 12:07:46.927845 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.926362    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134"} err="failed to get container status \"0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134\": rpc error: code = NotFound desc = could not find container \"0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134\": container with ID starting with 0ce5ab9fb05b9814938dfdb44206cc66a508977ae16667a86a9360d823c67134 not found: ID does not exist"
May 26 12:07:46.935599 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.935578    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"]
May 26 12:07:46.940293 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:46.940270    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-6gfcp"]
May 26 12:07:47.906735 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:47.906696    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" event={"ID":"ac9867c6-0656-43f4-a6bc-ec8723f075d6","Type":"ContainerStarted","Data":"db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910"}
May 26 12:07:47.906735 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:47.906732    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" event={"ID":"ac9867c6-0656-43f4-a6bc-ec8723f075d6","Type":"ContainerStarted","Data":"487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93"}
May 26 12:07:47.907297 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:47.907052    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:47.907297 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:47.907162    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:47.908646 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:47.908617    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:07:47.930802 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:47.930748    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podStartSLOduration=5.930737118 podStartE2EDuration="5.930737118s" podCreationTimestamp="2026-05-26 12:07:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:07:47.92844127 +0000 UTC m=+2430.291390939" watchObservedRunningTime="2026-05-26 12:07:47.930737118 +0000 UTC m=+2430.293686786"
May 26 12:07:48.241603 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:48.241526    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" path="/var/lib/kubelet/pods/e212988e-e671-4e0c-acdc-a04a86197d96/volumes"
May 26 12:07:48.909547 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:48.909510    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:07:53.913599 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:53.913570    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:07:53.914180 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:07:53.914155    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:08:01.406471 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:08:01.406438    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:08:01.409449 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:08:01.409428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:08:03.914109 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:08:03.914069    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:08:13.914325 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:08:13.914284    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:08:23.914634 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:08:23.914594    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:08:33.914123 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:08:33.914072    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:08:43.914416 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:08:43.914374    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:08:53.914243 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:08:53.914203    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:09:03.914954 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:03.914927    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:09:12.234713 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.234639    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"]
May 26 12:09:12.235093 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.234978    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" containerID="cri-o://487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93" gracePeriod=30
May 26 12:09:12.235093 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.235026    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kube-rbac-proxy" containerID="cri-o://db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910" gracePeriod=30
May 26 12:09:12.387337 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387302    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"]
May 26 12:09:12.387558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387547    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="storage-initializer"
May 26 12:09:12.387598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="storage-initializer"
May 26 12:09:12.387598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387570    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container"
May 26 12:09:12.387598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387576    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container"
May 26 12:09:12.387598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387588    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kube-rbac-proxy"
May 26 12:09:12.387598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387594    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kube-rbac-proxy"
May 26 12:09:12.387753 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387644    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kserve-container"
May 26 12:09:12.387753 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.387651    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e212988e-e671-4e0c-acdc-a04a86197d96" containerName="kube-rbac-proxy"
May 26 12:09:12.390531 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.390515    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.392704 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.392683    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
May 26 12:09:12.392838 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.392683    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
May 26 12:09:12.400106 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.400077    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"]
May 26 12:09:12.429652 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.429614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.429788 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.429687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a93e083-b30d-4530-90d7-d507972afb2b-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.429788 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.429718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxzqz\" (UniqueName: \"kubernetes.io/projected/6a93e083-b30d-4530-90d7-d507972afb2b-kube-api-access-pxzqz\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.429788 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.429746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a93e083-b30d-4530-90d7-d507972afb2b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.530990 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.530910    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a93e083-b30d-4530-90d7-d507972afb2b-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.530990 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.530946    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxzqz\" (UniqueName: \"kubernetes.io/projected/6a93e083-b30d-4530-90d7-d507972afb2b-kube-api-access-pxzqz\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.530990 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.530969    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a93e083-b30d-4530-90d7-d507972afb2b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.531230 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.531011    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.531230 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:09:12.531162    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
May 26 12:09:12.531230 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:09:12.531222    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls podName:6a93e083-b30d-4530-90d7-d507972afb2b nodeName:}" failed. No retries permitted until 2026-05-26 12:09:13.031203256 +0000 UTC m=+2515.394152910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls") pod "isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" (UID: "6a93e083-b30d-4530-90d7-d507972afb2b") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
May 26 12:09:12.531454 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.531436    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a93e083-b30d-4530-90d7-d507972afb2b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.531599 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.531581    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a93e083-b30d-4530-90d7-d507972afb2b-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:12.543138 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:12.543110    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxzqz\" (UniqueName: \"kubernetes.io/projected/6a93e083-b30d-4530-90d7-d507972afb2b-kube-api-access-pxzqz\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:13.035155 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.035119    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:13.037715 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.037690    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:13.125750 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.125718    2563 generic.go:358] "Generic (PLEG): container finished" podID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerID="db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910" exitCode=2
May 26 12:09:13.125920 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.125766    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" event={"ID":"ac9867c6-0656-43f4-a6bc-ec8723f075d6","Type":"ContainerDied","Data":"db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910"}
May 26 12:09:13.301326 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.301291    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:13.428347 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.428278    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"]
May 26 12:09:13.430812 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:09:13.430764    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a93e083_b30d_4530_90d7_d507972afb2b.slice/crio-45970f9c3791184fd1a9558926f6e1454e49449b4431df44e3ee973b83b2a952 WatchSource:0}: Error finding container 45970f9c3791184fd1a9558926f6e1454e49449b4431df44e3ee973b83b2a952: Status 404 returned error can't find the container with id 45970f9c3791184fd1a9558926f6e1454e49449b4431df44e3ee973b83b2a952
May 26 12:09:13.432938 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.432920    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:09:13.910700 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.910660    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 26 12:09:13.914664 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:13.914628    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 12:09:14.130055 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:14.130015    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" event={"ID":"6a93e083-b30d-4530-90d7-d507972afb2b","Type":"ContainerStarted","Data":"20b251bedf2560c109ea830db8a76bd573e20b70d41d410ab8175c9952ea1890"}
May 26 12:09:14.130055 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:14.130061    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" event={"ID":"6a93e083-b30d-4530-90d7-d507972afb2b","Type":"ContainerStarted","Data":"45970f9c3791184fd1a9558926f6e1454e49449b4431df44e3ee973b83b2a952"}
May 26 12:09:16.871607 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.871583    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:09:16.962595 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.962520    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kserve-provision-location\") pod \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") "
May 26 12:09:16.962595 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.962560    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9867c6-0656-43f4-a6bc-ec8723f075d6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") "
May 26 12:09:16.962595 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.962583    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9867c6-0656-43f4-a6bc-ec8723f075d6-proxy-tls\") pod \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") "
May 26 12:09:16.962807 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.962632    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ch6b\" (UniqueName: \"kubernetes.io/projected/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kube-api-access-5ch6b\") pod \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\" (UID: \"ac9867c6-0656-43f4-a6bc-ec8723f075d6\") "
May 26 12:09:16.962865 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.962841    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac9867c6-0656-43f4-a6bc-ec8723f075d6" (UID: "ac9867c6-0656-43f4-a6bc-ec8723f075d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:09:16.962949 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.962926    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9867c6-0656-43f4-a6bc-ec8723f075d6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "ac9867c6-0656-43f4-a6bc-ec8723f075d6" (UID: "ac9867c6-0656-43f4-a6bc-ec8723f075d6"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:09:16.964847 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.964814    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kube-api-access-5ch6b" (OuterVolumeSpecName: "kube-api-access-5ch6b") pod "ac9867c6-0656-43f4-a6bc-ec8723f075d6" (UID: "ac9867c6-0656-43f4-a6bc-ec8723f075d6"). InnerVolumeSpecName "kube-api-access-5ch6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:09:16.964847 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:16.964840    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9867c6-0656-43f4-a6bc-ec8723f075d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ac9867c6-0656-43f4-a6bc-ec8723f075d6" (UID: "ac9867c6-0656-43f4-a6bc-ec8723f075d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:09:17.063495 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.063461    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:09:17.063495 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.063490    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9867c6-0656-43f4-a6bc-ec8723f075d6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:09:17.063495 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.063501    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9867c6-0656-43f4-a6bc-ec8723f075d6-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:09:17.063705 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.063510    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ch6b\" (UniqueName: \"kubernetes.io/projected/ac9867c6-0656-43f4-a6bc-ec8723f075d6-kube-api-access-5ch6b\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:09:17.139394 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.139360    2563 generic.go:358] "Generic (PLEG): container finished" podID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerID="487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93" exitCode=0
May 26 12:09:17.139555 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.139399    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" event={"ID":"ac9867c6-0656-43f4-a6bc-ec8723f075d6","Type":"ContainerDied","Data":"487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93"}
May 26 12:09:17.139555 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.139429    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9" event={"ID":"ac9867c6-0656-43f4-a6bc-ec8723f075d6","Type":"ContainerDied","Data":"8cf40c8cc248b6b701ca4b8dcd423e81c977aee85f0a5cb433d4c882c02117fe"}
May 26 12:09:17.139555 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.139438    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"
May 26 12:09:17.139555 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.139448    2563 scope.go:117] "RemoveContainer" containerID="db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910"
May 26 12:09:17.147439 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.147422    2563 scope.go:117] "RemoveContainer" containerID="487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93"
May 26 12:09:17.154211 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.154192    2563 scope.go:117] "RemoveContainer" containerID="3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc"
May 26 12:09:17.160386 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.160370    2563 scope.go:117] "RemoveContainer" containerID="db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910"
May 26 12:09:17.160625 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:09:17.160608    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910\": container with ID starting with db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910 not found: ID does not exist" containerID="db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910"
May 26 12:09:17.160664 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.160633    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910"} err="failed to get container status \"db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910\": rpc error: code = NotFound desc = could not find container \"db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910\": container with ID starting with db87d02594ce3d1693c9bd3cb8b9fd4e9885c7124802a2d3e6fdaef34ee93910 not found: ID does not exist"
May 26 12:09:17.160664 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.160650    2563 scope.go:117] "RemoveContainer" containerID="487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93"
May 26 12:09:17.160888 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:09:17.160872    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93\": container with ID starting with 487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93 not found: ID does not exist" containerID="487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93"
May 26 12:09:17.160931 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.160893    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93"} err="failed to get container status \"487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93\": rpc error: code = NotFound desc = could not find container \"487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93\": container with ID starting with 487a208b17bcd7f20b538454a0d1e44a8edc154c2133452630755f68c306da93 not found: ID does not exist"
May 26 12:09:17.160931 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.160906    2563 scope.go:117] "RemoveContainer" containerID="3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc"
May 26 12:09:17.161091 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:09:17.161074    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc\": container with ID starting with 3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc not found: ID does not exist" containerID="3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc"
May 26 12:09:17.161125 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.161098    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc"} err="failed to get container status \"3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc\": rpc error: code = NotFound desc = could not find container \"3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc\": container with ID starting with 3cc1e966135d5bb8411924b445120c5cc4fd554a15160355b9bc807df6f061cc not found: ID does not exist"
May 26 12:09:17.163017 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.162996    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"]
May 26 12:09:17.175635 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:17.175616    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-d8ck9"]
May 26 12:09:18.143856 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:18.143826    2563 generic.go:358] "Generic (PLEG): container finished" podID="6a93e083-b30d-4530-90d7-d507972afb2b" containerID="20b251bedf2560c109ea830db8a76bd573e20b70d41d410ab8175c9952ea1890" exitCode=0
May 26 12:09:18.144236 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:18.143902    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" event={"ID":"6a93e083-b30d-4530-90d7-d507972afb2b","Type":"ContainerDied","Data":"20b251bedf2560c109ea830db8a76bd573e20b70d41d410ab8175c9952ea1890"}
May 26 12:09:18.241948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:18.241919    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" path="/var/lib/kubelet/pods/ac9867c6-0656-43f4-a6bc-ec8723f075d6/volumes"
May 26 12:09:19.148516 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:19.148486    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" event={"ID":"6a93e083-b30d-4530-90d7-d507972afb2b","Type":"ContainerStarted","Data":"ec34e7ef0a4c5b1cb9d3e969c38616ee886d6a77da6d938fb1c04d650aaa81d9"}
May 26 12:09:19.148516 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:19.148523    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" event={"ID":"6a93e083-b30d-4530-90d7-d507972afb2b","Type":"ContainerStarted","Data":"38517edd76c7cdeb88d898a8e8d5c6a6894ed3334fbdbd3196ce377b0dbe7ee3"}
May 26 12:09:19.148955 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:19.148723    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:19.169446 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:19.169390    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podStartSLOduration=7.169374412 podStartE2EDuration="7.169374412s" podCreationTimestamp="2026-05-26 12:09:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:09:19.167281006 +0000 UTC m=+2521.530230675" watchObservedRunningTime="2026-05-26 12:09:19.169374412 +0000 UTC m=+2521.532324083"
May 26 12:09:20.151602 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:20.151571    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:20.152837 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:20.152806    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:09:21.154047 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:21.154007    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:09:26.158390 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:26.158360    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:09:26.158882 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:26.158856    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:09:36.158997 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:36.158956    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:09:46.159489 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:46.159452    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:09:56.158908 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:09:56.158870    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:10:06.159426 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:06.159386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:10:16.159851 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:16.159786    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:10:26.159031 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:26.158988    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:10:36.159897 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:36.159821    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:10:42.580169 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.580135    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"]
May 26 12:10:42.580583 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.580553    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" containerID="cri-o://38517edd76c7cdeb88d898a8e8d5c6a6894ed3334fbdbd3196ce377b0dbe7ee3" gracePeriod=30
May 26 12:10:42.580726 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.580640    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kube-rbac-proxy" containerID="cri-o://ec34e7ef0a4c5b1cb9d3e969c38616ee886d6a77da6d938fb1c04d650aaa81d9" gracePeriod=30
May 26 12:10:42.729410 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729373    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"]
May 26 12:10:42.729658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729644    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="storage-initializer"
May 26 12:10:42.729708 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729661    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="storage-initializer"
May 26 12:10:42.729708 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729673    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kube-rbac-proxy"
May 26 12:10:42.729708 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729679    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kube-rbac-proxy"
May 26 12:10:42.729708 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729698    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container"
May 26 12:10:42.729708 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729704    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container"
May 26 12:10:42.729923 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729744    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kube-rbac-proxy"
May 26 12:10:42.729923 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.729752    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="ac9867c6-0656-43f4-a6bc-ec8723f075d6" containerName="kserve-container"
May 26 12:10:42.732629 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.732611    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.734887 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.734865    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
May 26 12:10:42.734981 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.734958    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
May 26 12:10:42.741515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.741491    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"]
May 26 12:10:42.780013 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.779972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.780182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.780036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.780182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.780068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgjc\" (UniqueName: \"kubernetes.io/projected/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kube-api-access-vfgjc\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.780182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.780092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.880659 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.880574    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.880659 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.880621    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgjc\" (UniqueName: \"kubernetes.io/projected/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kube-api-access-vfgjc\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.880906 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.880743    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.880906 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.880836    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.880983 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:10:42.880916    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
May 26 12:10:42.881022 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:10:42.881006    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls podName:6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09 nodeName:}" failed. No retries permitted until 2026-05-26 12:10:43.380983959 +0000 UTC m=+2605.743933612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" (UID: "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
May 26 12:10:42.881164 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.881148    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.881340 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.881324    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:42.889344 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:42.889324    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgjc\" (UniqueName: \"kubernetes.io/projected/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kube-api-access-vfgjc\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:43.363834 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:43.363784    2563 generic.go:358] "Generic (PLEG): container finished" podID="6a93e083-b30d-4530-90d7-d507972afb2b" containerID="ec34e7ef0a4c5b1cb9d3e969c38616ee886d6a77da6d938fb1c04d650aaa81d9" exitCode=2
May 26 12:10:43.363834 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:43.363824    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" event={"ID":"6a93e083-b30d-4530-90d7-d507972afb2b","Type":"ContainerDied","Data":"ec34e7ef0a4c5b1cb9d3e969c38616ee886d6a77da6d938fb1c04d650aaa81d9"}
May 26 12:10:43.384240 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:43.384211    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:43.386857 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:43.386826    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:43.643850 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:43.643749    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:43.769740 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:43.769709    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"]
May 26 12:10:43.773307 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:10:43.773277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7e4c3d_c09b_47cf_b6b7_86eeadb3cc09.slice/crio-c4790e0ae717166077e141c7b8f03cad7ec766ee44fdf65c631bd48360c29eea WatchSource:0}: Error finding container c4790e0ae717166077e141c7b8f03cad7ec766ee44fdf65c631bd48360c29eea: Status 404 returned error can't find the container with id c4790e0ae717166077e141c7b8f03cad7ec766ee44fdf65c631bd48360c29eea
May 26 12:10:44.368006 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:44.367961    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" event={"ID":"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09","Type":"ContainerStarted","Data":"3fd930e3e4560d71c8ce3cd8f384f7defa284ef141be4742a96f654745431f36"}
May 26 12:10:44.368006 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:44.368009    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" event={"ID":"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09","Type":"ContainerStarted","Data":"c4790e0ae717166077e141c7b8f03cad7ec766ee44fdf65c631bd48360c29eea"}
May 26 12:10:46.154288 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:46.154245    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 26 12:10:46.159547 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:46.159504    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 12:10:48.378834 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.378781    2563 generic.go:358] "Generic (PLEG): container finished" podID="6a93e083-b30d-4530-90d7-d507972afb2b" containerID="38517edd76c7cdeb88d898a8e8d5c6a6894ed3334fbdbd3196ce377b0dbe7ee3" exitCode=0
May 26 12:10:48.379211 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.378843    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" event={"ID":"6a93e083-b30d-4530-90d7-d507972afb2b","Type":"ContainerDied","Data":"38517edd76c7cdeb88d898a8e8d5c6a6894ed3334fbdbd3196ce377b0dbe7ee3"}
May 26 12:10:48.380087 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.380063    2563 generic.go:358] "Generic (PLEG): container finished" podID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerID="3fd930e3e4560d71c8ce3cd8f384f7defa284ef141be4742a96f654745431f36" exitCode=0
May 26 12:10:48.380182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.380131    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" event={"ID":"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09","Type":"ContainerDied","Data":"3fd930e3e4560d71c8ce3cd8f384f7defa284ef141be4742a96f654745431f36"}
May 26 12:10:48.442632 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.442610    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:10:48.525397 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.525376    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a93e083-b30d-4530-90d7-d507972afb2b-kserve-provision-location\") pod \"6a93e083-b30d-4530-90d7-d507972afb2b\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") "
May 26 12:10:48.525525 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.525412    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a93e083-b30d-4530-90d7-d507972afb2b-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"6a93e083-b30d-4530-90d7-d507972afb2b\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") "
May 26 12:10:48.525525 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.525438    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls\") pod \"6a93e083-b30d-4530-90d7-d507972afb2b\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") "
May 26 12:10:48.525525 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.525462    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxzqz\" (UniqueName: \"kubernetes.io/projected/6a93e083-b30d-4530-90d7-d507972afb2b-kube-api-access-pxzqz\") pod \"6a93e083-b30d-4530-90d7-d507972afb2b\" (UID: \"6a93e083-b30d-4530-90d7-d507972afb2b\") "
May 26 12:10:48.525763 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.525736    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a93e083-b30d-4530-90d7-d507972afb2b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a93e083-b30d-4530-90d7-d507972afb2b" (UID: "6a93e083-b30d-4530-90d7-d507972afb2b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:48.525838 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.525759    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a93e083-b30d-4530-90d7-d507972afb2b-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "6a93e083-b30d-4530-90d7-d507972afb2b" (UID: "6a93e083-b30d-4530-90d7-d507972afb2b"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:10:48.527515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.527489    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a93e083-b30d-4530-90d7-d507972afb2b" (UID: "6a93e083-b30d-4530-90d7-d507972afb2b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:10:48.527673 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.527651    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a93e083-b30d-4530-90d7-d507972afb2b-kube-api-access-pxzqz" (OuterVolumeSpecName: "kube-api-access-pxzqz") pod "6a93e083-b30d-4530-90d7-d507972afb2b" (UID: "6a93e083-b30d-4530-90d7-d507972afb2b"). InnerVolumeSpecName "kube-api-access-pxzqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:10:48.626013 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.625931    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a93e083-b30d-4530-90d7-d507972afb2b-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:10:48.626013 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.625964    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a93e083-b30d-4530-90d7-d507972afb2b-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:10:48.626013 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.625976    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a93e083-b30d-4530-90d7-d507972afb2b-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:10:48.626013 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:48.625987    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxzqz\" (UniqueName: \"kubernetes.io/projected/6a93e083-b30d-4530-90d7-d507972afb2b-kube-api-access-pxzqz\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:10:49.384683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.384644    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls" event={"ID":"6a93e083-b30d-4530-90d7-d507972afb2b","Type":"ContainerDied","Data":"45970f9c3791184fd1a9558926f6e1454e49449b4431df44e3ee973b83b2a952"}
May 26 12:10:49.384683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.384685    2563 scope.go:117] "RemoveContainer" containerID="ec34e7ef0a4c5b1cb9d3e969c38616ee886d6a77da6d938fb1c04d650aaa81d9"
May 26 12:10:49.385232 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.384683    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"
May 26 12:10:49.386858 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.386831    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" event={"ID":"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09","Type":"ContainerStarted","Data":"3d4e300de2822b854af709bfe73f67dd3c280b5947b074de51153c75b3709a25"}
May 26 12:10:49.386956 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.386870    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" event={"ID":"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09","Type":"ContainerStarted","Data":"76d39cf2d85d51a13ec6b7ff684e8de026f59185f2bf9e90a287ca030110f2dc"}
May 26 12:10:49.387063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.387042    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:49.387110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.387077    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:10:49.393063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.392956    2563 scope.go:117] "RemoveContainer" containerID="38517edd76c7cdeb88d898a8e8d5c6a6894ed3334fbdbd3196ce377b0dbe7ee3"
May 26 12:10:49.401938 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.401914    2563 scope.go:117] "RemoveContainer" containerID="20b251bedf2560c109ea830db8a76bd573e20b70d41d410ab8175c9952ea1890"
May 26 12:10:49.408391 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.408350    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podStartSLOduration=7.408336283 podStartE2EDuration="7.408336283s" podCreationTimestamp="2026-05-26 12:10:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:10:49.407387867 +0000 UTC m=+2611.770337536" watchObservedRunningTime="2026-05-26 12:10:49.408336283 +0000 UTC m=+2611.771285952"
May 26 12:10:49.422449 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.422422    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"]
May 26 12:10:49.430721 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:49.430700    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-f97ls"]
May 26 12:10:50.242567 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:50.242533    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" path="/var/lib/kubelet/pods/6a93e083-b30d-4530-90d7-d507972afb2b/volumes"
May 26 12:10:55.396403 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:10:55.396371    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:11:25.397598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:11:25.397560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
May 26 12:11:35.397218 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:11:35.397179    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
May 26 12:11:45.397193 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:11:45.397156    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
May 26 12:11:55.397384 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:11:55.397344    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
May 26 12:12:05.401143 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:05.401104    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:12:12.791026 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.790995    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"]
May 26 12:12:12.791490 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.791409    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container" containerID="cri-o://76d39cf2d85d51a13ec6b7ff684e8de026f59185f2bf9e90a287ca030110f2dc" gracePeriod=30
May 26 12:12:12.791558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.791458    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kube-rbac-proxy" containerID="cri-o://3d4e300de2822b854af709bfe73f67dd3c280b5947b074de51153c75b3709a25" gracePeriod=30
May 26 12:12:12.973007 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.972977    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"]
May 26 12:12:12.973297 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.973282    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kube-rbac-proxy"
May 26 12:12:12.973408 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.973300    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kube-rbac-proxy"
May 26 12:12:12.973408 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.973314    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="storage-initializer"
May 26 12:12:12.973408 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.973323    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="storage-initializer"
May 26 12:12:12.973408 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.973342    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container"
May 26 12:12:12.973408 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.973350    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container"
May 26 12:12:12.973668 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.973417    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kube-rbac-proxy"
May 26 12:12:12.973668 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.973435    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a93e083-b30d-4530-90d7-d507972afb2b" containerName="kserve-container"
May 26 12:12:12.976319 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.976299    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:12.978237 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.978204    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
May 26 12:12:12.978237 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.978213    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
May 26 12:12:12.986509 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:12.986486    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"]
May 26 12:12:13.039953 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.039928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6ffc64-aad7-4001-884e-5269b0b60f46-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.040124 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.039960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6ffc64-aad7-4001-884e-5269b0b60f46-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.040124 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.040005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dg7q\" (UniqueName: \"kubernetes.io/projected/2e6ffc64-aad7-4001-884e-5269b0b60f46-kube-api-access-6dg7q\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.040124 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.040076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6ffc64-aad7-4001-884e-5269b0b60f46-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.141016 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.140926    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6ffc64-aad7-4001-884e-5269b0b60f46-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.141016 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.140964    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6ffc64-aad7-4001-884e-5269b0b60f46-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.141257 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.141211    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6dg7q\" (UniqueName: \"kubernetes.io/projected/2e6ffc64-aad7-4001-884e-5269b0b60f46-kube-api-access-6dg7q\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.141347 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.141286    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6ffc64-aad7-4001-884e-5269b0b60f46-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.141680 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.141650    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6ffc64-aad7-4001-884e-5269b0b60f46-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.141870 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.141853    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6ffc64-aad7-4001-884e-5269b0b60f46-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.143731 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.143708    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6ffc64-aad7-4001-884e-5269b0b60f46-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.149945 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.149925    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dg7q\" (UniqueName: \"kubernetes.io/projected/2e6ffc64-aad7-4001-884e-5269b0b60f46-kube-api-access-6dg7q\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.285959 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.285924    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:13.417451 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.417363    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"]
May 26 12:12:13.420295 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:12:13.420263    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6ffc64_aad7_4001_884e_5269b0b60f46.slice/crio-6943475ba361d8228cedc77c07284f5e36c4cc26451d205b03a59e28859bab82 WatchSource:0}: Error finding container 6943475ba361d8228cedc77c07284f5e36c4cc26451d205b03a59e28859bab82: Status 404 returned error can't find the container with id 6943475ba361d8228cedc77c07284f5e36c4cc26451d205b03a59e28859bab82
May 26 12:12:13.608236 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.608195    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" event={"ID":"2e6ffc64-aad7-4001-884e-5269b0b60f46","Type":"ContainerStarted","Data":"c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf"}
May 26 12:12:13.608236 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.608240    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" event={"ID":"2e6ffc64-aad7-4001-884e-5269b0b60f46","Type":"ContainerStarted","Data":"6943475ba361d8228cedc77c07284f5e36c4cc26451d205b03a59e28859bab82"}
May 26 12:12:13.610167 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.610142    2563 generic.go:358] "Generic (PLEG): container finished" podID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerID="3d4e300de2822b854af709bfe73f67dd3c280b5947b074de51153c75b3709a25" exitCode=2
May 26 12:12:13.610293 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:13.610215    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" event={"ID":"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09","Type":"ContainerDied","Data":"3d4e300de2822b854af709bfe73f67dd3c280b5947b074de51153c75b3709a25"}
May 26 12:12:15.391876 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:15.391836    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 26 12:12:15.397238 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:15.397208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
May 26 12:12:17.620889 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.620858    2563 generic.go:358] "Generic (PLEG): container finished" podID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerID="c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf" exitCode=0
May 26 12:12:17.621278 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.620916    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" event={"ID":"2e6ffc64-aad7-4001-884e-5269b0b60f46","Type":"ContainerDied","Data":"c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf"}
May 26 12:12:17.622956 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.622934    2563 generic.go:358] "Generic (PLEG): container finished" podID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerID="76d39cf2d85d51a13ec6b7ff684e8de026f59185f2bf9e90a287ca030110f2dc" exitCode=0
May 26 12:12:17.623066 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.622969    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" event={"ID":"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09","Type":"ContainerDied","Data":"76d39cf2d85d51a13ec6b7ff684e8de026f59185f2bf9e90a287ca030110f2dc"}
May 26 12:12:17.623066 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.622991    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj" event={"ID":"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09","Type":"ContainerDied","Data":"c4790e0ae717166077e141c7b8f03cad7ec766ee44fdf65c631bd48360c29eea"}
May 26 12:12:17.623066 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.623003    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4790e0ae717166077e141c7b8f03cad7ec766ee44fdf65c631bd48360c29eea"
May 26 12:12:17.631082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.631066    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:12:17.674077 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.674055    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfgjc\" (UniqueName: \"kubernetes.io/projected/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kube-api-access-vfgjc\") pod \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") "
May 26 12:12:17.674168 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.674102    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls\") pod \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") "
May 26 12:12:17.674168 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.674138    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") "
May 26 12:12:17.674284 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.674175    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kserve-provision-location\") pod \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\" (UID: \"6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09\") "
May 26 12:12:17.674597 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.674563    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" (UID: "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:12:17.674597 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.674585    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" (UID: "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:12:17.676497 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.676475    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" (UID: "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:12:17.676627 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.676609    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kube-api-access-vfgjc" (OuterVolumeSpecName: "kube-api-access-vfgjc") pod "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" (UID: "6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09"). InnerVolumeSpecName "kube-api-access-vfgjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:12:17.775273 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.775212    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgjc\" (UniqueName: \"kubernetes.io/projected/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kube-api-access-vfgjc\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:12:17.775273 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.775240    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:12:17.775273 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.775251    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:12:17.775273 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:17.775260    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:12:18.627423 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:18.627389    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" event={"ID":"2e6ffc64-aad7-4001-884e-5269b0b60f46","Type":"ContainerStarted","Data":"31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8"}
May 26 12:12:18.627423 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:18.627429    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" event={"ID":"2e6ffc64-aad7-4001-884e-5269b0b60f46","Type":"ContainerStarted","Data":"8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae"}
May 26 12:12:18.627871 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:18.627404    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"
May 26 12:12:18.627921 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:18.627908    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:18.652919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:18.652871    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podStartSLOduration=6.652860498 podStartE2EDuration="6.652860498s" podCreationTimestamp="2026-05-26 12:12:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:12:18.651936223 +0000 UTC m=+2701.014885892" watchObservedRunningTime="2026-05-26 12:12:18.652860498 +0000 UTC m=+2701.015810167"
May 26 12:12:18.665200 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:18.665174    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"]
May 26 12:12:18.669908 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:18.669886    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-rxqlj"]
May 26 12:12:19.630842 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:19.630810    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:20.241907 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:20.241871    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" path="/var/lib/kubelet/pods/6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09/volumes"
May 26 12:12:25.639092 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:25.639062    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:12:55.639736 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:12:55.639686    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
May 26 12:13:01.425099 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:01.425066    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:13:01.428692 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:01.428672    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:13:05.639904 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:05.639868    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
May 26 12:13:15.640619 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:15.640576    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
May 26 12:13:25.640402 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:25.640356    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
May 26 12:13:29.239078 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:29.239039    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
May 26 12:13:39.242354 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:39.242260    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:13:42.997433 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:42.997400    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"]
May 26 12:13:42.997837 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:42.997725    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container" containerID="cri-o://8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae" gracePeriod=30
May 26 12:13:42.997837 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:42.997763    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kube-rbac-proxy" containerID="cri-o://31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8" gracePeriod=30
May 26 12:13:43.126529 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126497    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"]
May 26 12:13:43.126761 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126749    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="storage-initializer"
May 26 12:13:43.126829 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126762    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="storage-initializer"
May 26 12:13:43.126829 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126777    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container"
May 26 12:13:43.126829 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126783    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container"
May 26 12:13:43.126829 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126809    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kube-rbac-proxy"
May 26 12:13:43.126829 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126816    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kube-rbac-proxy"
May 26 12:13:43.127029 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126872    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kserve-container"
May 26 12:13:43.127029 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.126880    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a7e4c3d-c09b-47cf-b6b7-86eeadb3cc09" containerName="kube-rbac-proxy"
May 26 12:13:43.129591 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.129571    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.132062 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.132043    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
May 26 12:13:43.132338 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.132322    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
May 26 12:13:43.142139 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.142113    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"]
May 26 12:13:43.188222 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.188192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbn7\" (UniqueName: \"kubernetes.io/projected/770c2a37-b5a2-4813-bada-2e3f3f673167-kube-api-access-2vbn7\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.188367 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.188240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770c2a37-b5a2-4813-bada-2e3f3f673167-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.188367 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.188300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770c2a37-b5a2-4813-bada-2e3f3f673167-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.188367 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.188322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770c2a37-b5a2-4813-bada-2e3f3f673167-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.289080 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.288994    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbn7\" (UniqueName: \"kubernetes.io/projected/770c2a37-b5a2-4813-bada-2e3f3f673167-kube-api-access-2vbn7\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.289080 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.289041    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770c2a37-b5a2-4813-bada-2e3f3f673167-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.289342 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.289086    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770c2a37-b5a2-4813-bada-2e3f3f673167-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.289342 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.289107    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770c2a37-b5a2-4813-bada-2e3f3f673167-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.289480 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.289461    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770c2a37-b5a2-4813-bada-2e3f3f673167-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.289735 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.289716    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770c2a37-b5a2-4813-bada-2e3f3f673167-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.291849 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.291828    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770c2a37-b5a2-4813-bada-2e3f3f673167-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.297559 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.297539    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbn7\" (UniqueName: \"kubernetes.io/projected/770c2a37-b5a2-4813-bada-2e3f3f673167-kube-api-access-2vbn7\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.439730 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.439694    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:43.572416 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.568003    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"]
May 26 12:13:43.844907 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.844809    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" event={"ID":"770c2a37-b5a2-4813-bada-2e3f3f673167","Type":"ContainerStarted","Data":"3f82f4c49deb779da5c6b8301c56c3852c230a1fb0ebe781dfa11a18ae736a2f"}
May 26 12:13:43.844907 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.844856    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" event={"ID":"770c2a37-b5a2-4813-bada-2e3f3f673167","Type":"ContainerStarted","Data":"29349a5412ca3e500b0202f237737a6608fb2b5bb7c620c3d8b5d3fed979ae3e"}
May 26 12:13:43.846671 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.846642    2563 generic.go:358] "Generic (PLEG): container finished" podID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerID="31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8" exitCode=2
May 26 12:13:43.846769 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:43.846683    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" event={"ID":"2e6ffc64-aad7-4001-884e-5269b0b60f46","Type":"ContainerDied","Data":"31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8"}
May 26 12:13:45.633670 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:45.633630    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
May 26 12:13:47.643524 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.643498    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:13:47.718440 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.718410    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6ffc64-aad7-4001-884e-5269b0b60f46-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"2e6ffc64-aad7-4001-884e-5269b0b60f46\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") "
May 26 12:13:47.718575 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.718463    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6ffc64-aad7-4001-884e-5269b0b60f46-kserve-provision-location\") pod \"2e6ffc64-aad7-4001-884e-5269b0b60f46\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") "
May 26 12:13:47.718575 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.718490    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6ffc64-aad7-4001-884e-5269b0b60f46-proxy-tls\") pod \"2e6ffc64-aad7-4001-884e-5269b0b60f46\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") "
May 26 12:13:47.718575 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.718512    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dg7q\" (UniqueName: \"kubernetes.io/projected/2e6ffc64-aad7-4001-884e-5269b0b60f46-kube-api-access-6dg7q\") pod \"2e6ffc64-aad7-4001-884e-5269b0b60f46\" (UID: \"2e6ffc64-aad7-4001-884e-5269b0b60f46\") "
May 26 12:13:47.718830 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.718786    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6ffc64-aad7-4001-884e-5269b0b60f46-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "2e6ffc64-aad7-4001-884e-5269b0b60f46" (UID: "2e6ffc64-aad7-4001-884e-5269b0b60f46"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:13:47.718895 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.718814    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6ffc64-aad7-4001-884e-5269b0b60f46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e6ffc64-aad7-4001-884e-5269b0b60f46" (UID: "2e6ffc64-aad7-4001-884e-5269b0b60f46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:13:47.720665 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.720640    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6ffc64-aad7-4001-884e-5269b0b60f46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e6ffc64-aad7-4001-884e-5269b0b60f46" (UID: "2e6ffc64-aad7-4001-884e-5269b0b60f46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:13:47.720815 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.720775    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6ffc64-aad7-4001-884e-5269b0b60f46-kube-api-access-6dg7q" (OuterVolumeSpecName: "kube-api-access-6dg7q") pod "2e6ffc64-aad7-4001-884e-5269b0b60f46" (UID: "2e6ffc64-aad7-4001-884e-5269b0b60f46"). InnerVolumeSpecName "kube-api-access-6dg7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:13:47.819436 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.819404    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e6ffc64-aad7-4001-884e-5269b0b60f46-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:13:47.819436 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.819435    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e6ffc64-aad7-4001-884e-5269b0b60f46-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:13:47.819605 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.819445    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e6ffc64-aad7-4001-884e-5269b0b60f46-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:13:47.819605 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.819454    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dg7q\" (UniqueName: \"kubernetes.io/projected/2e6ffc64-aad7-4001-884e-5269b0b60f46-kube-api-access-6dg7q\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:13:47.858731 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.858659    2563 generic.go:358] "Generic (PLEG): container finished" podID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerID="3f82f4c49deb779da5c6b8301c56c3852c230a1fb0ebe781dfa11a18ae736a2f" exitCode=0
May 26 12:13:47.858852 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.858736    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" event={"ID":"770c2a37-b5a2-4813-bada-2e3f3f673167","Type":"ContainerDied","Data":"3f82f4c49deb779da5c6b8301c56c3852c230a1fb0ebe781dfa11a18ae736a2f"}
May 26 12:13:47.860337 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.860312    2563 generic.go:358] "Generic (PLEG): container finished" podID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerID="8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae" exitCode=0
May 26 12:13:47.860408 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.860361    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" event={"ID":"2e6ffc64-aad7-4001-884e-5269b0b60f46","Type":"ContainerDied","Data":"8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae"}
May 26 12:13:47.860408 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.860386    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"
May 26 12:13:47.860408 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.860393    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn" event={"ID":"2e6ffc64-aad7-4001-884e-5269b0b60f46","Type":"ContainerDied","Data":"6943475ba361d8228cedc77c07284f5e36c4cc26451d205b03a59e28859bab82"}
May 26 12:13:47.860515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.860410    2563 scope.go:117] "RemoveContainer" containerID="31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8"
May 26 12:13:47.868400 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.868373    2563 scope.go:117] "RemoveContainer" containerID="8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae"
May 26 12:13:47.875035 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.875017    2563 scope.go:117] "RemoveContainer" containerID="c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf"
May 26 12:13:47.882127 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.882107    2563 scope.go:117] "RemoveContainer" containerID="31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8"
May 26 12:13:47.882424 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:13:47.882401    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8\": container with ID starting with 31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8 not found: ID does not exist" containerID="31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8"
May 26 12:13:47.882522 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.882429    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8"} err="failed to get container status \"31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8\": rpc error: code = NotFound desc = could not find container \"31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8\": container with ID starting with 31bd87f8d04dc0febfea9d52f80f830e57f0adaacab1f42c94ec613c65f69af8 not found: ID does not exist"
May 26 12:13:47.882522 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.882446    2563 scope.go:117] "RemoveContainer" containerID="8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae"
May 26 12:13:47.882710 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:13:47.882689    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae\": container with ID starting with 8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae not found: ID does not exist" containerID="8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae"
May 26 12:13:47.882781 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.882715    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae"} err="failed to get container status \"8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae\": rpc error: code = NotFound desc = could not find container \"8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae\": container with ID starting with 8ae9eae37f62228838907b5a3b4c25a7fde3490a41f209e0415cc91fc5b1e6ae not found: ID does not exist"
May 26 12:13:47.882781 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.882731    2563 scope.go:117] "RemoveContainer" containerID="c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf"
May 26 12:13:47.882997 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:13:47.882980    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf\": container with ID starting with c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf not found: ID does not exist" containerID="c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf"
May 26 12:13:47.883040 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.883004    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf"} err="failed to get container status \"c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf\": rpc error: code = NotFound desc = could not find container \"c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf\": container with ID starting with c444bb44ff9620097edbc6d31739fff4b1cb31b12e955dee7bfd268138829bcf not found: ID does not exist"
May 26 12:13:47.890294 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.890275    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"]
May 26 12:13:47.895464 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:47.895446    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-fwwcn"]
May 26 12:13:48.242121 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:48.242044    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" path="/var/lib/kubelet/pods/2e6ffc64-aad7-4001-884e-5269b0b60f46/volumes"
May 26 12:13:48.865073 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:48.865028    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" event={"ID":"770c2a37-b5a2-4813-bada-2e3f3f673167","Type":"ContainerStarted","Data":"6d63c2c68ffaa5708f90150b64707cc3b949126ae2ef5459ae8e061aa4e898cb"}
May 26 12:13:48.865073 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:48.865073    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" event={"ID":"770c2a37-b5a2-4813-bada-2e3f3f673167","Type":"ContainerStarted","Data":"f3243aa710770837f3dfd4ca6390841c429cd20d86ece165b8d8b48b33be1ca8"}
May 26 12:13:48.865598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:48.865347    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:48.865598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:48.865410    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:13:48.885668 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:48.885630    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" podStartSLOduration=5.885617783 podStartE2EDuration="5.885617783s" podCreationTimestamp="2026-05-26 12:13:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:13:48.884362801 +0000 UTC m=+2791.247312470" watchObservedRunningTime="2026-05-26 12:13:48.885617783 +0000 UTC m=+2791.248567452"
May 26 12:13:54.874178 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:13:54.874143    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:14:24.875317 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:14:24.875273    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 12:14:34.875214 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:14:34.875172    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 12:14:44.875558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:14:44.875513    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 12:14:54.875699 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:14:54.875657    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 12:15:01.242512 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:01.242480    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:15:03.220026 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:03.219978    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"]
May 26 12:15:03.220483 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:03.220386    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kserve-container" containerID="cri-o://f3243aa710770837f3dfd4ca6390841c429cd20d86ece165b8d8b48b33be1ca8" gracePeriod=30
May 26 12:15:03.220483 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:03.220437    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kube-rbac-proxy" containerID="cri-o://6d63c2c68ffaa5708f90150b64707cc3b949126ae2ef5459ae8e061aa4e898cb" gracePeriod=30
May 26 12:15:04.074677 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:04.074644    2563 generic.go:358] "Generic (PLEG): container finished" podID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerID="6d63c2c68ffaa5708f90150b64707cc3b949126ae2ef5459ae8e061aa4e898cb" exitCode=2
May 26 12:15:04.074873 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:04.074691    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" event={"ID":"770c2a37-b5a2-4813-bada-2e3f3f673167","Type":"ContainerDied","Data":"6d63c2c68ffaa5708f90150b64707cc3b949126ae2ef5459ae8e061aa4e898cb"}
May 26 12:15:04.868940 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:04.868902    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
May 26 12:15:05.417481 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417446    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"]
May 26 12:15:05.417695 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417684    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="storage-initializer"
May 26 12:15:05.417741 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417697    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="storage-initializer"
May 26 12:15:05.417741 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417710    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container"
May 26 12:15:05.417741 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container"
May 26 12:15:05.417741 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417728    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kube-rbac-proxy"
May 26 12:15:05.417741 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417734    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kube-rbac-proxy"
May 26 12:15:05.417935 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417775    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kube-rbac-proxy"
May 26 12:15:05.417935 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.417782    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="2e6ffc64-aad7-4001-884e-5269b0b60f46" containerName="kserve-container"
May 26 12:15:05.420543 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.420526    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.422560 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.422541    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
May 26 12:15:05.423015 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.422997    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
May 26 12:15:05.434188 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.434168    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"]
May 26 12:15:05.584693 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.584606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzbh\" (UniqueName: \"kubernetes.io/projected/71995f58-75c5-4824-a73c-c8093e011f9a-kube-api-access-2zzbh\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.584693 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.584660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71995f58-75c5-4824-a73c-c8093e011f9a-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.584693 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.584679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71995f58-75c5-4824-a73c-c8093e011f9a-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.584952 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.584777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.685948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.685851    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzbh\" (UniqueName: \"kubernetes.io/projected/71995f58-75c5-4824-a73c-c8093e011f9a-kube-api-access-2zzbh\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.685948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.685909    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71995f58-75c5-4824-a73c-c8093e011f9a-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.686174 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.686031    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71995f58-75c5-4824-a73c-c8093e011f9a-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.686174 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.686135    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.686263 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:15:05.686247    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
May 26 12:15:05.686317 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:15:05.686309    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls podName:71995f58-75c5-4824-a73c-c8093e011f9a nodeName:}" failed. No retries permitted until 2026-05-26 12:15:06.186289356 +0000 UTC m=+2868.549239014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls") pod "isvc-sklearn-predictor-bc9cf9b57-rvz5m" (UID: "71995f58-75c5-4824-a73c-c8093e011f9a") : secret "isvc-sklearn-predictor-serving-cert" not found
May 26 12:15:05.686436 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.686414    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71995f58-75c5-4824-a73c-c8093e011f9a-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.686562 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.686547    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71995f58-75c5-4824-a73c-c8093e011f9a-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:05.694431 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:05.694409    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzbh\" (UniqueName: \"kubernetes.io/projected/71995f58-75c5-4824-a73c-c8093e011f9a-kube-api-access-2zzbh\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:06.190686 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:06.190575    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:06.202072 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:06.193099    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-rvz5m\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:06.330061 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:06.329993    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:06.470089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:06.469987    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"]
May 26 12:15:06.473498 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:15:06.473472    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71995f58_75c5_4824_a73c_c8093e011f9a.slice/crio-80534a8c3c5cd1774253895593377d6b42e3a01f4c2a52f5d783978cf9b9049f WatchSource:0}: Error finding container 80534a8c3c5cd1774253895593377d6b42e3a01f4c2a52f5d783978cf9b9049f: Status 404 returned error can't find the container with id 80534a8c3c5cd1774253895593377d6b42e3a01f4c2a52f5d783978cf9b9049f
May 26 12:15:06.475430 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:06.475413    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:15:07.083912 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:07.083870    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" event={"ID":"71995f58-75c5-4824-a73c-c8093e011f9a","Type":"ContainerStarted","Data":"253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e"}
May 26 12:15:07.083912 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:07.083914    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" event={"ID":"71995f58-75c5-4824-a73c-c8093e011f9a","Type":"ContainerStarted","Data":"80534a8c3c5cd1774253895593377d6b42e3a01f4c2a52f5d783978cf9b9049f"}
May 26 12:15:09.091577 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.091541    2563 generic.go:358] "Generic (PLEG): container finished" podID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerID="f3243aa710770837f3dfd4ca6390841c429cd20d86ece165b8d8b48b33be1ca8" exitCode=0
May 26 12:15:09.092009 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.091619    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" event={"ID":"770c2a37-b5a2-4813-bada-2e3f3f673167","Type":"ContainerDied","Data":"f3243aa710770837f3dfd4ca6390841c429cd20d86ece165b8d8b48b33be1ca8"}
May 26 12:15:09.159276 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.159251    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:15:09.318175 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.318137    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770c2a37-b5a2-4813-bada-2e3f3f673167-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"770c2a37-b5a2-4813-bada-2e3f3f673167\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") "
May 26 12:15:09.318380 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.318207    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770c2a37-b5a2-4813-bada-2e3f3f673167-proxy-tls\") pod \"770c2a37-b5a2-4813-bada-2e3f3f673167\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") "
May 26 12:15:09.318380 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.318239    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vbn7\" (UniqueName: \"kubernetes.io/projected/770c2a37-b5a2-4813-bada-2e3f3f673167-kube-api-access-2vbn7\") pod \"770c2a37-b5a2-4813-bada-2e3f3f673167\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") "
May 26 12:15:09.318380 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.318268    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770c2a37-b5a2-4813-bada-2e3f3f673167-kserve-provision-location\") pod \"770c2a37-b5a2-4813-bada-2e3f3f673167\" (UID: \"770c2a37-b5a2-4813-bada-2e3f3f673167\") "
May 26 12:15:09.318560 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.318517    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770c2a37-b5a2-4813-bada-2e3f3f673167-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "770c2a37-b5a2-4813-bada-2e3f3f673167" (UID: "770c2a37-b5a2-4813-bada-2e3f3f673167"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:15:09.318626 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.318596    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770c2a37-b5a2-4813-bada-2e3f3f673167-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "770c2a37-b5a2-4813-bada-2e3f3f673167" (UID: "770c2a37-b5a2-4813-bada-2e3f3f673167"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:15:09.320533 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.320509    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770c2a37-b5a2-4813-bada-2e3f3f673167-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "770c2a37-b5a2-4813-bada-2e3f3f673167" (UID: "770c2a37-b5a2-4813-bada-2e3f3f673167"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:15:09.320636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.320508    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770c2a37-b5a2-4813-bada-2e3f3f673167-kube-api-access-2vbn7" (OuterVolumeSpecName: "kube-api-access-2vbn7") pod "770c2a37-b5a2-4813-bada-2e3f3f673167" (UID: "770c2a37-b5a2-4813-bada-2e3f3f673167"). InnerVolumeSpecName "kube-api-access-2vbn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:15:09.419559 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.419503    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770c2a37-b5a2-4813-bada-2e3f3f673167-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:15:09.419559 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.419553    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vbn7\" (UniqueName: \"kubernetes.io/projected/770c2a37-b5a2-4813-bada-2e3f3f673167-kube-api-access-2vbn7\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:15:09.419559 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.419565    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770c2a37-b5a2-4813-bada-2e3f3f673167-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:15:09.419559 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:09.419575    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770c2a37-b5a2-4813-bada-2e3f3f673167-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:15:10.097890 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:10.097853    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79" event={"ID":"770c2a37-b5a2-4813-bada-2e3f3f673167","Type":"ContainerDied","Data":"29349a5412ca3e500b0202f237737a6608fb2b5bb7c620c3d8b5d3fed979ae3e"}
May 26 12:15:10.097890 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:10.097896    2563 scope.go:117] "RemoveContainer" containerID="6d63c2c68ffaa5708f90150b64707cc3b949126ae2ef5459ae8e061aa4e898cb"
May 26 12:15:10.098412 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:10.097911    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"
May 26 12:15:10.105436 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:10.105416    2563 scope.go:117] "RemoveContainer" containerID="f3243aa710770837f3dfd4ca6390841c429cd20d86ece165b8d8b48b33be1ca8"
May 26 12:15:10.112430 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:10.112415    2563 scope.go:117] "RemoveContainer" containerID="3f82f4c49deb779da5c6b8301c56c3852c230a1fb0ebe781dfa11a18ae736a2f"
May 26 12:15:10.128202 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:10.128180    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"]
May 26 12:15:10.134425 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:10.134403    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-hcg79"]
May 26 12:15:10.242553 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:10.242524    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" path="/var/lib/kubelet/pods/770c2a37-b5a2-4813-bada-2e3f3f673167/volumes"
May 26 12:15:11.103090 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:11.103059    2563 generic.go:358] "Generic (PLEG): container finished" podID="71995f58-75c5-4824-a73c-c8093e011f9a" containerID="253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e" exitCode=0
May 26 12:15:11.103485 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:11.103111    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" event={"ID":"71995f58-75c5-4824-a73c-c8093e011f9a","Type":"ContainerDied","Data":"253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e"}
May 26 12:15:12.108300 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:12.108266    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" event={"ID":"71995f58-75c5-4824-a73c-c8093e011f9a","Type":"ContainerStarted","Data":"a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460"}
May 26 12:15:12.108682 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:12.108308    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" event={"ID":"71995f58-75c5-4824-a73c-c8093e011f9a","Type":"ContainerStarted","Data":"1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851"}
May 26 12:15:12.108682 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:12.108511    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:12.132185 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:12.132139    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podStartSLOduration=7.132125121 podStartE2EDuration="7.132125121s" podCreationTimestamp="2026-05-26 12:15:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:15:12.129286296 +0000 UTC m=+2874.492235963" watchObservedRunningTime="2026-05-26 12:15:12.132125121 +0000 UTC m=+2874.495074790"
May 26 12:15:13.111636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:13.111601    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:13.112885 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:13.112859    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:15:14.114685 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:14.114643    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:15:19.119406 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:19.119380    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:15:19.119900 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:19.119870    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:15:29.120665 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:29.120627    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:15:39.119915 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:39.119876    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:15:49.120384 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:49.120340    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:15:59.120853 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:15:59.120812    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:16:09.120828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:09.120768    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:16:19.120751 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:19.120719    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:16:25.525310 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.525278    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"]
May 26 12:16:25.525702 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.525565    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" containerID="cri-o://1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851" gracePeriod=30
May 26 12:16:25.525766 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.525692    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kube-rbac-proxy" containerID="cri-o://a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460" gracePeriod=30
May 26 12:16:25.642744 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.642716    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"]
May 26 12:16:25.643019 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.643006    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kube-rbac-proxy"
May 26 12:16:25.643072 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.643021    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kube-rbac-proxy"
May 26 12:16:25.643072 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.643035    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kserve-container"
May 26 12:16:25.643072 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.643042    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kserve-container"
May 26 12:16:25.643072 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.643050    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="storage-initializer"
May 26 12:16:25.643072 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.643056    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="storage-initializer"
May 26 12:16:25.643226 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.643100    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kserve-container"
May 26 12:16:25.643226 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.643109    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="770c2a37-b5a2-4813-bada-2e3f3f673167" containerName="kube-rbac-proxy"
May 26 12:16:25.645966 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.645947    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.648371 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.648353    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
May 26 12:16:25.648485 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.648468    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
May 26 12:16:25.660565 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.660542    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"]
May 26 12:16:25.748073 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.748040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.748265 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.748093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.748265 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.748111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrdx\" (UniqueName: \"kubernetes.io/projected/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kube-api-access-klrdx\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.748265 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.748173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.849348 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.849311    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.849348 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.849349    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-klrdx\" (UniqueName: \"kubernetes.io/projected/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kube-api-access-klrdx\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.849606 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.849367    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.849606 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.849423    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.849868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.849843    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.850096 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.850074    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.851990 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.851964    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.859559 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.859536    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klrdx\" (UniqueName: \"kubernetes.io/projected/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kube-api-access-klrdx\") pod \"sklearn-v2-mlserver-predictor-65d8664766-smn5t\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:25.955385 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:25.955355    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:26.082766 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:26.082717    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"]
May 26 12:16:26.085949 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:16:26.085919    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee0fa8_924f_44b3_947c_4d1e213bb2c2.slice/crio-325e94a4aa549085d75d0844971d847f89bca48fc3d7d2ffe065041c2795b361 WatchSource:0}: Error finding container 325e94a4aa549085d75d0844971d847f89bca48fc3d7d2ffe065041c2795b361: Status 404 returned error can't find the container with id 325e94a4aa549085d75d0844971d847f89bca48fc3d7d2ffe065041c2795b361
May 26 12:16:26.301013 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:26.300978    2563 generic.go:358] "Generic (PLEG): container finished" podID="71995f58-75c5-4824-a73c-c8093e011f9a" containerID="a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460" exitCode=2
May 26 12:16:26.301204 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:26.301044    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" event={"ID":"71995f58-75c5-4824-a73c-c8093e011f9a","Type":"ContainerDied","Data":"a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460"}
May 26 12:16:26.302378 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:26.302351    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" event={"ID":"aaee0fa8-924f-44b3-947c-4d1e213bb2c2","Type":"ContainerStarted","Data":"f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b"}
May 26 12:16:26.302378 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:26.302381    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" event={"ID":"aaee0fa8-924f-44b3-947c-4d1e213bb2c2","Type":"ContainerStarted","Data":"325e94a4aa549085d75d0844971d847f89bca48fc3d7d2ffe065041c2795b361"}
May 26 12:16:29.115134 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:29.115089    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
May 26 12:16:29.120444 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:29.120421    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 12:16:29.971432 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:29.971408    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:16:30.084329 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.084287    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zzbh\" (UniqueName: \"kubernetes.io/projected/71995f58-75c5-4824-a73c-c8093e011f9a-kube-api-access-2zzbh\") pod \"71995f58-75c5-4824-a73c-c8093e011f9a\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") "
May 26 12:16:30.084493 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.084347    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71995f58-75c5-4824-a73c-c8093e011f9a-kserve-provision-location\") pod \"71995f58-75c5-4824-a73c-c8093e011f9a\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") "
May 26 12:16:30.084493 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.084370    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls\") pod \"71995f58-75c5-4824-a73c-c8093e011f9a\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") "
May 26 12:16:30.084493 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.084408    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71995f58-75c5-4824-a73c-c8093e011f9a-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"71995f58-75c5-4824-a73c-c8093e011f9a\" (UID: \"71995f58-75c5-4824-a73c-c8093e011f9a\") "
May 26 12:16:30.084713 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.084696    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71995f58-75c5-4824-a73c-c8093e011f9a-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "71995f58-75c5-4824-a73c-c8093e011f9a" (UID: "71995f58-75c5-4824-a73c-c8093e011f9a"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:16:30.084757 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.084697    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71995f58-75c5-4824-a73c-c8093e011f9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71995f58-75c5-4824-a73c-c8093e011f9a" (UID: "71995f58-75c5-4824-a73c-c8093e011f9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:16:30.086598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.086578    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71995f58-75c5-4824-a73c-c8093e011f9a-kube-api-access-2zzbh" (OuterVolumeSpecName: "kube-api-access-2zzbh") pod "71995f58-75c5-4824-a73c-c8093e011f9a" (UID: "71995f58-75c5-4824-a73c-c8093e011f9a"). InnerVolumeSpecName "kube-api-access-2zzbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:16:30.086667 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.086606    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71995f58-75c5-4824-a73c-c8093e011f9a" (UID: "71995f58-75c5-4824-a73c-c8093e011f9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:16:30.185164 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.185131    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71995f58-75c5-4824-a73c-c8093e011f9a-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:16:30.185164 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.185159    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zzbh\" (UniqueName: \"kubernetes.io/projected/71995f58-75c5-4824-a73c-c8093e011f9a-kube-api-access-2zzbh\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:16:30.185164 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.185172    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71995f58-75c5-4824-a73c-c8093e011f9a-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:16:30.185583 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.185182    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71995f58-75c5-4824-a73c-c8093e011f9a-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:16:30.314849 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.314786    2563 generic.go:358] "Generic (PLEG): container finished" podID="71995f58-75c5-4824-a73c-c8093e011f9a" containerID="1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851" exitCode=0
May 26 12:16:30.315082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.314869    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" event={"ID":"71995f58-75c5-4824-a73c-c8093e011f9a","Type":"ContainerDied","Data":"1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851"}
May 26 12:16:30.315082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.314896    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"
May 26 12:16:30.315082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.314904    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m" event={"ID":"71995f58-75c5-4824-a73c-c8093e011f9a","Type":"ContainerDied","Data":"80534a8c3c5cd1774253895593377d6b42e3a01f4c2a52f5d783978cf9b9049f"}
May 26 12:16:30.315082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.314920    2563 scope.go:117] "RemoveContainer" containerID="a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460"
May 26 12:16:30.316351 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.316328    2563 generic.go:358] "Generic (PLEG): container finished" podID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerID="f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b" exitCode=0
May 26 12:16:30.316451 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.316379    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" event={"ID":"aaee0fa8-924f-44b3-947c-4d1e213bb2c2","Type":"ContainerDied","Data":"f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b"}
May 26 12:16:30.322750 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.322735    2563 scope.go:117] "RemoveContainer" containerID="1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851"
May 26 12:16:30.329552 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.329532    2563 scope.go:117] "RemoveContainer" containerID="253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e"
May 26 12:16:30.336899 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.336878    2563 scope.go:117] "RemoveContainer" containerID="a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460"
May 26 12:16:30.337133 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:16:30.337115    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460\": container with ID starting with a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460 not found: ID does not exist" containerID="a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460"
May 26 12:16:30.337185 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.337140    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460"} err="failed to get container status \"a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460\": rpc error: code = NotFound desc = could not find container \"a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460\": container with ID starting with a90304a31ba6be93c6c77906c0e23bc91a2a0850ea40bdbc80fdd298b7001460 not found: ID does not exist"
May 26 12:16:30.337185 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.337160    2563 scope.go:117] "RemoveContainer" containerID="1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851"
May 26 12:16:30.337401 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:16:30.337383    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851\": container with ID starting with 1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851 not found: ID does not exist" containerID="1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851"
May 26 12:16:30.337453 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.337406    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851"} err="failed to get container status \"1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851\": rpc error: code = NotFound desc = could not find container \"1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851\": container with ID starting with 1e1087531ac7e098c9bda41a5809e31974b8ff29988b49bec08bc30853a48851 not found: ID does not exist"
May 26 12:16:30.337453 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.337421    2563 scope.go:117] "RemoveContainer" containerID="253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e"
May 26 12:16:30.337606 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:16:30.337591    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e\": container with ID starting with 253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e not found: ID does not exist" containerID="253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e"
May 26 12:16:30.337646 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.337608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e"} err="failed to get container status \"253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e\": rpc error: code = NotFound desc = could not find container \"253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e\": container with ID starting with 253d023f71942367da8ac41200cd71a9d2681ccd124b769ac6d3f6f37641603e not found: ID does not exist"
May 26 12:16:30.351968 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.351947    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"]
May 26 12:16:30.356165 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:30.356143    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-rvz5m"]
May 26 12:16:31.321683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:31.321648    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" event={"ID":"aaee0fa8-924f-44b3-947c-4d1e213bb2c2","Type":"ContainerStarted","Data":"821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2"}
May 26 12:16:31.321683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:31.321685    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" event={"ID":"aaee0fa8-924f-44b3-947c-4d1e213bb2c2","Type":"ContainerStarted","Data":"57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd"}
May 26 12:16:31.322108 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:31.321937    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:31.343115 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:31.343065    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" podStartSLOduration=6.343051906 podStartE2EDuration="6.343051906s" podCreationTimestamp="2026-05-26 12:16:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:16:31.341953975 +0000 UTC m=+2953.704903640" watchObservedRunningTime="2026-05-26 12:16:31.343051906 +0000 UTC m=+2953.706001573"
May 26 12:16:32.242515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:32.242482    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" path="/var/lib/kubelet/pods/71995f58-75c5-4824-a73c-c8093e011f9a/volumes"
May 26 12:16:32.324060 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:32.324034    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:16:38.332480 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:16:38.332405    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:17:08.401155 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:08.401111    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 26 12:17:18.335229 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:18.335199    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:17:20.993919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:20.993888    2563 scope.go:117] "RemoveContainer" containerID="3d4e300de2822b854af709bfe73f67dd3c280b5947b074de51153c75b3709a25"
May 26 12:17:21.001535 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:21.001515    2563 scope.go:117] "RemoveContainer" containerID="3fd930e3e4560d71c8ce3cd8f384f7defa284ef141be4742a96f654745431f36"
May 26 12:17:21.007825 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:21.007806    2563 scope.go:117] "RemoveContainer" containerID="76d39cf2d85d51a13ec6b7ff684e8de026f59185f2bf9e90a287ca030110f2dc"
May 26 12:17:25.679257 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.679219    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"]
May 26 12:17:25.679706 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.679614    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kserve-container" containerID="cri-o://57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd" gracePeriod=30
May 26 12:17:25.679775 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.679711    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kube-rbac-proxy" containerID="cri-o://821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2" gracePeriod=30
May 26 12:17:25.797338 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797308    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"]
May 26 12:17:25.797592 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797579    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kube-rbac-proxy"
May 26 12:17:25.797636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kube-rbac-proxy"
May 26 12:17:25.797636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797603    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container"
May 26 12:17:25.797636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797608    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container"
May 26 12:17:25.797636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797625    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="storage-initializer"
May 26 12:17:25.797636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797633    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="storage-initializer"
May 26 12:17:25.797806 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797674    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kube-rbac-proxy"
May 26 12:17:25.797806 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.797681    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="71995f58-75c5-4824-a73c-c8093e011f9a" containerName="kserve-container"
May 26 12:17:25.800435 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.800415    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.802875 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.802852    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
May 26 12:17:25.802978 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.802862    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
May 26 12:17:25.813560 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.813536    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"]
May 26 12:17:25.897961 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.897928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83dd6908-8040-446a-b275-f567752f4b98-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.898110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.897976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83dd6908-8040-446a-b275-f567752f4b98-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.898110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.898003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk94s\" (UniqueName: \"kubernetes.io/projected/83dd6908-8040-446a-b275-f567752f4b98-kube-api-access-mk94s\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.898110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.898059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83dd6908-8040-446a-b275-f567752f4b98-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.998697 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.998616    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83dd6908-8040-446a-b275-f567752f4b98-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.998697 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.998658    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83dd6908-8040-446a-b275-f567752f4b98-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.998962 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.998717    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83dd6908-8040-446a-b275-f567752f4b98-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.998962 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.998860    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mk94s\" (UniqueName: \"kubernetes.io/projected/83dd6908-8040-446a-b275-f567752f4b98-kube-api-access-mk94s\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.999163 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.999141    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83dd6908-8040-446a-b275-f567752f4b98-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:25.999403 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:25.999380    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83dd6908-8040-446a-b275-f567752f4b98-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:26.001454 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:26.001430    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83dd6908-8040-446a-b275-f567752f4b98-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:26.007783 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:26.007766    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk94s\" (UniqueName: \"kubernetes.io/projected/83dd6908-8040-446a-b275-f567752f4b98-kube-api-access-mk94s\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:26.110769 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:26.110738    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:26.242125 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:17:26.242078    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83dd6908_8040_446a_b275_f567752f4b98.slice/crio-7ae9218b3ce8665a52f2fe87c15a9644a3e08c9671d2d69fe99ade913f96c3b5 WatchSource:0}: Error finding container 7ae9218b3ce8665a52f2fe87c15a9644a3e08c9671d2d69fe99ade913f96c3b5: Status 404 returned error can't find the container with id 7ae9218b3ce8665a52f2fe87c15a9644a3e08c9671d2d69fe99ade913f96c3b5
May 26 12:17:26.242725 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:26.242704    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"]
May 26 12:17:26.461766 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:26.461728    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" event={"ID":"83dd6908-8040-446a-b275-f567752f4b98","Type":"ContainerStarted","Data":"c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d"}
May 26 12:17:26.461766 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:26.461772    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" event={"ID":"83dd6908-8040-446a-b275-f567752f4b98","Type":"ContainerStarted","Data":"7ae9218b3ce8665a52f2fe87c15a9644a3e08c9671d2d69fe99ade913f96c3b5"}
May 26 12:17:26.463676 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:26.463652    2563 generic.go:358] "Generic (PLEG): container finished" podID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerID="821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2" exitCode=2
May 26 12:17:26.463806 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:26.463722    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" event={"ID":"aaee0fa8-924f-44b3-947c-4d1e213bb2c2","Type":"ContainerDied","Data":"821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2"}
May 26 12:17:28.327653 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:28.327612    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
May 26 12:17:32.481901 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:32.481861    2563 generic.go:358] "Generic (PLEG): container finished" podID="83dd6908-8040-446a-b275-f567752f4b98" containerID="c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d" exitCode=0
May 26 12:17:32.482293 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:32.481933    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" event={"ID":"83dd6908-8040-446a-b275-f567752f4b98","Type":"ContainerDied","Data":"c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d"}
May 26 12:17:33.024440 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.024418    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:17:33.046326 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.046302    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kserve-provision-location\") pod \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") "
May 26 12:17:33.046461 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.046334    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") "
May 26 12:17:33.046461 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.046357    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-proxy-tls\") pod \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") "
May 26 12:17:33.046677 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.046647    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "aaee0fa8-924f-44b3-947c-4d1e213bb2c2" (UID: "aaee0fa8-924f-44b3-947c-4d1e213bb2c2"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:17:33.046677 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.046660    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aaee0fa8-924f-44b3-947c-4d1e213bb2c2" (UID: "aaee0fa8-924f-44b3-947c-4d1e213bb2c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:17:33.048919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.048873    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aaee0fa8-924f-44b3-947c-4d1e213bb2c2" (UID: "aaee0fa8-924f-44b3-947c-4d1e213bb2c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:17:33.147062 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.147022    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klrdx\" (UniqueName: \"kubernetes.io/projected/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kube-api-access-klrdx\") pod \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\" (UID: \"aaee0fa8-924f-44b3-947c-4d1e213bb2c2\") "
May 26 12:17:33.147226 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.147210    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:17:33.147297 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.147235    2563 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:17:33.147297 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.147252    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:17:33.149093 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.149068    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kube-api-access-klrdx" (OuterVolumeSpecName: "kube-api-access-klrdx") pod "aaee0fa8-924f-44b3-947c-4d1e213bb2c2" (UID: "aaee0fa8-924f-44b3-947c-4d1e213bb2c2"). InnerVolumeSpecName "kube-api-access-klrdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:17:33.247632 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.247598    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klrdx\" (UniqueName: \"kubernetes.io/projected/aaee0fa8-924f-44b3-947c-4d1e213bb2c2-kube-api-access-klrdx\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:17:33.486581 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.486496    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" event={"ID":"83dd6908-8040-446a-b275-f567752f4b98","Type":"ContainerStarted","Data":"aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446"}
May 26 12:17:33.486581 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.486540    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" event={"ID":"83dd6908-8040-446a-b275-f567752f4b98","Type":"ContainerStarted","Data":"c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d"}
May 26 12:17:33.487084 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.486780    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:33.487084 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.486829    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:33.488202 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.488175    2563 generic.go:358] "Generic (PLEG): container finished" podID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerID="57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd" exitCode=0
May 26 12:17:33.488313 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.488249    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" event={"ID":"aaee0fa8-924f-44b3-947c-4d1e213bb2c2","Type":"ContainerDied","Data":"57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd"}
May 26 12:17:33.488313 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.488274    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t" event={"ID":"aaee0fa8-924f-44b3-947c-4d1e213bb2c2","Type":"ContainerDied","Data":"325e94a4aa549085d75d0844971d847f89bca48fc3d7d2ffe065041c2795b361"}
May 26 12:17:33.488313 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.488252    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"
May 26 12:17:33.488313 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.488296    2563 scope.go:117] "RemoveContainer" containerID="821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2"
May 26 12:17:33.488516 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.488363    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 26 12:17:33.496832 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.496814    2563 scope.go:117] "RemoveContainer" containerID="57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd"
May 26 12:17:33.504080 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.504054    2563 scope.go:117] "RemoveContainer" containerID="f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b"
May 26 12:17:33.507490 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.507444    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" podStartSLOduration=8.507428916 podStartE2EDuration="8.507428916s" podCreationTimestamp="2026-05-26 12:17:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:17:33.506145362 +0000 UTC m=+3015.869095031" watchObservedRunningTime="2026-05-26 12:17:33.507428916 +0000 UTC m=+3015.870378586"
May 26 12:17:33.511434 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.511412    2563 scope.go:117] "RemoveContainer" containerID="821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2"
May 26 12:17:33.511707 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:17:33.511687    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2\": container with ID starting with 821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2 not found: ID does not exist" containerID="821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2"
May 26 12:17:33.511779 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.511713    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2"} err="failed to get container status \"821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2\": rpc error: code = NotFound desc = could not find container \"821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2\": container with ID starting with 821069df25d0711e212a111c82609a4e31772ae2fd78b7d24884f7b5bd4e39e2 not found: ID does not exist"
May 26 12:17:33.511779 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.511734    2563 scope.go:117] "RemoveContainer" containerID="57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd"
May 26 12:17:33.512018 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:17:33.511993    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd\": container with ID starting with 57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd not found: ID does not exist" containerID="57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd"
May 26 12:17:33.512082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.512026    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd"} err="failed to get container status \"57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd\": rpc error: code = NotFound desc = could not find container \"57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd\": container with ID starting with 57bf27a47de1bca8871789b7cc1a2db2aab026b8d2106f2883a672d0694f3bdd not found: ID does not exist"
May 26 12:17:33.512082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.512045    2563 scope.go:117] "RemoveContainer" containerID="f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b"
May 26 12:17:33.512325 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:17:33.512308    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b\": container with ID starting with f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b not found: ID does not exist" containerID="f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b"
May 26 12:17:33.512370 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.512339    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b"} err="failed to get container status \"f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b\": rpc error: code = NotFound desc = could not find container \"f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b\": container with ID starting with f2ba2b2981688a88944f1456c4044e3cf4dab1b26b508355e98cf981ee0c174b not found: ID does not exist"
May 26 12:17:33.518264 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.518245    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"]
May 26 12:17:33.523362 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:33.523342    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-smn5t"]
May 26 12:17:34.247005 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:34.246964    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" path="/var/lib/kubelet/pods/aaee0fa8-924f-44b3-947c-4d1e213bb2c2/volumes"
May 26 12:17:34.492089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:34.492044    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 26 12:17:39.496604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:39.496576    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:17:39.497112 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:39.497067    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
May 26 12:17:49.497713 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:17:49.497675    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:18:01.442311 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:01.442279    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:18:01.446384 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:01.446367    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:18:02.739499 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.739462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch_83dd6908-8040-446a-b275-f567752f4b98/kserve-container/0.log"
May 26 12:18:02.880097 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.880066    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"]
May 26 12:18:02.880473 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.880429    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kserve-container" containerID="cri-o://c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d" gracePeriod=30
May 26 12:18:02.880571 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.880520    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kube-rbac-proxy" containerID="cri-o://aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446" gracePeriod=30
May 26 12:18:02.970395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970364    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"]
May 26 12:18:02.970720 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970705    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="storage-initializer"
May 26 12:18:02.970720 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970722    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="storage-initializer"
May 26 12:18:02.970868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970736    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kube-rbac-proxy"
May 26 12:18:02.970868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970741    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kube-rbac-proxy"
May 26 12:18:02.970868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970749    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kserve-container"
May 26 12:18:02.970868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970758    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kserve-container"
May 26 12:18:02.970868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970816    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kserve-container"
May 26 12:18:02.970868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.970826    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="aaee0fa8-924f-44b3-947c-4d1e213bb2c2" containerName="kube-rbac-proxy"
May 26 12:18:02.973764 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.973749    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:02.975751 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.975731    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
May 26 12:18:02.975751 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.975741    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
May 26 12:18:02.984031 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:02.984007    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"]
May 26 12:18:03.054339 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.054305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9tj\" (UniqueName: \"kubernetes.io/projected/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kube-api-access-9g9tj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.054339 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.054340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.054532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.054384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.054532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.054409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.155263 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.155228    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9tj\" (UniqueName: \"kubernetes.io/projected/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kube-api-access-9g9tj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.155263 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.155269    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.155483 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.155299    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.155483 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.155320    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.155483 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:18:03.155397    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
May 26 12:18:03.155483 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:18:03.155468    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls podName:ed11f405-8cfd-4ecd-82a8-2c32ce88eeba nodeName:}" failed. No retries permitted until 2026-05-26 12:18:03.655446223 +0000 UTC m=+3046.018395871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" (UID: "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
May 26 12:18:03.155713 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.155697    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.155957 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.155940    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.163222 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.163196    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9tj\" (UniqueName: \"kubernetes.io/projected/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kube-api-access-9g9tj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.580021 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.579980    2563 generic.go:358] "Generic (PLEG): container finished" podID="83dd6908-8040-446a-b275-f567752f4b98" containerID="aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446" exitCode=2
May 26 12:18:03.580230 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.580077    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" event={"ID":"83dd6908-8040-446a-b275-f567752f4b98","Type":"ContainerDied","Data":"aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446"}
May 26 12:18:03.659664 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.659628    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.662301 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.662274    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:03.884082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:03.883970    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:04.018533 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.018511    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:18:04.027920 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.027894    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"]
May 26 12:18:04.030987 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:18:04.030966    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded11f405_8cfd_4ecd_82a8_2c32ce88eeba.slice/crio-439a6ba17b162070706b174c96cc72ef29520150736e16437c890516471c1c9e WatchSource:0}: Error finding container 439a6ba17b162070706b174c96cc72ef29520150736e16437c890516471c1c9e: Status 404 returned error can't find the container with id 439a6ba17b162070706b174c96cc72ef29520150736e16437c890516471c1c9e
May 26 12:18:04.062639 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.062610    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk94s\" (UniqueName: \"kubernetes.io/projected/83dd6908-8040-446a-b275-f567752f4b98-kube-api-access-mk94s\") pod \"83dd6908-8040-446a-b275-f567752f4b98\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") "
May 26 12:18:04.062767 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.062670    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83dd6908-8040-446a-b275-f567752f4b98-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"83dd6908-8040-446a-b275-f567752f4b98\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") "
May 26 12:18:04.062767 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.062732    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83dd6908-8040-446a-b275-f567752f4b98-proxy-tls\") pod \"83dd6908-8040-446a-b275-f567752f4b98\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") "
May 26 12:18:04.062961 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.062777    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83dd6908-8040-446a-b275-f567752f4b98-kserve-provision-location\") pod \"83dd6908-8040-446a-b275-f567752f4b98\" (UID: \"83dd6908-8040-446a-b275-f567752f4b98\") "
May 26 12:18:04.063069 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.063018    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83dd6908-8040-446a-b275-f567752f4b98-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "83dd6908-8040-446a-b275-f567752f4b98" (UID: "83dd6908-8040-446a-b275-f567752f4b98"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:18:04.065303 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.065270    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83dd6908-8040-446a-b275-f567752f4b98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83dd6908-8040-446a-b275-f567752f4b98" (UID: "83dd6908-8040-446a-b275-f567752f4b98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:18:04.065482 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.065464    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83dd6908-8040-446a-b275-f567752f4b98-kube-api-access-mk94s" (OuterVolumeSpecName: "kube-api-access-mk94s") pod "83dd6908-8040-446a-b275-f567752f4b98" (UID: "83dd6908-8040-446a-b275-f567752f4b98"). InnerVolumeSpecName "kube-api-access-mk94s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:18:04.095603 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.095573    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83dd6908-8040-446a-b275-f567752f4b98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83dd6908-8040-446a-b275-f567752f4b98" (UID: "83dd6908-8040-446a-b275-f567752f4b98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:18:04.163515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.163419    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83dd6908-8040-446a-b275-f567752f4b98-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:18:04.163515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.163450    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83dd6908-8040-446a-b275-f567752f4b98-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:18:04.163515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.163461    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83dd6908-8040-446a-b275-f567752f4b98-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:18:04.163515 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.163470    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk94s\" (UniqueName: \"kubernetes.io/projected/83dd6908-8040-446a-b275-f567752f4b98-kube-api-access-mk94s\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:18:04.583969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.583930    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" event={"ID":"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba","Type":"ContainerStarted","Data":"4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c"}
May 26 12:18:04.583969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.583967    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" event={"ID":"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba","Type":"ContainerStarted","Data":"439a6ba17b162070706b174c96cc72ef29520150736e16437c890516471c1c9e"}
May 26 12:18:04.585542 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.585519    2563 generic.go:358] "Generic (PLEG): container finished" podID="83dd6908-8040-446a-b275-f567752f4b98" containerID="c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d" exitCode=0
May 26 12:18:04.585673 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.585579    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" event={"ID":"83dd6908-8040-446a-b275-f567752f4b98","Type":"ContainerDied","Data":"c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d"}
May 26 12:18:04.585673 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.585597    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"
May 26 12:18:04.585673 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.585612    2563 scope.go:117] "RemoveContainer" containerID="aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446"
May 26 12:18:04.585860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.585602    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch" event={"ID":"83dd6908-8040-446a-b275-f567752f4b98","Type":"ContainerDied","Data":"7ae9218b3ce8665a52f2fe87c15a9644a3e08c9671d2d69fe99ade913f96c3b5"}
May 26 12:18:04.592830 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.592814    2563 scope.go:117] "RemoveContainer" containerID="c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d"
May 26 12:18:04.599370 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.599355    2563 scope.go:117] "RemoveContainer" containerID="c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d"
May 26 12:18:04.606365 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.606346    2563 scope.go:117] "RemoveContainer" containerID="aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446"
May 26 12:18:04.606616 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:18:04.606593    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446\": container with ID starting with aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446 not found: ID does not exist" containerID="aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446"
May 26 12:18:04.606718 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.606625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446"} err="failed to get container status \"aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446\": rpc error: code = NotFound desc = could not find container \"aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446\": container with ID starting with aa02eb37f1e88e7df76dbf73c5c78273d0c690f053f7146e3f4151a9605b3446 not found: ID does not exist"
May 26 12:18:04.606718 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.606643    2563 scope.go:117] "RemoveContainer" containerID="c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d"
May 26 12:18:04.606884 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:18:04.606867    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d\": container with ID starting with c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d not found: ID does not exist" containerID="c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d"
May 26 12:18:04.606934 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.606888    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d"} err="failed to get container status \"c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d\": rpc error: code = NotFound desc = could not find container \"c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d\": container with ID starting with c321200db83fc95fc9b20f30f9083c78b872f8043b90c3f9d93f5ffd11db7a4d not found: ID does not exist"
May 26 12:18:04.606934 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.606901    2563 scope.go:117] "RemoveContainer" containerID="c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d"
May 26 12:18:04.607139 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:18:04.607123    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d\": container with ID starting with c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d not found: ID does not exist" containerID="c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d"
May 26 12:18:04.607180 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.607143    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d"} err="failed to get container status \"c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d\": rpc error: code = NotFound desc = could not find container \"c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d\": container with ID starting with c8ee8decaa5b7776feab0f6ac7cef5113c72cd59bad656f523bf861609c40f9d not found: ID does not exist"
May 26 12:18:04.616911 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.616880    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"]
May 26 12:18:04.623855 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:04.623835    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-4hrch"]
May 26 12:18:06.241638 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:06.241547    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83dd6908-8040-446a-b275-f567752f4b98" path="/var/lib/kubelet/pods/83dd6908-8040-446a-b275-f567752f4b98/volumes"
May 26 12:18:08.598372 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:08.598338    2563 generic.go:358] "Generic (PLEG): container finished" podID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerID="4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c" exitCode=0
May 26 12:18:08.598755 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:08.598412    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" event={"ID":"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba","Type":"ContainerDied","Data":"4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c"}
May 26 12:18:09.603738 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:09.603706    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" event={"ID":"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba","Type":"ContainerStarted","Data":"dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348"}
May 26 12:18:09.604182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:09.603746    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" event={"ID":"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba","Type":"ContainerStarted","Data":"f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34"}
May 26 12:18:09.604182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:09.603981    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:09.621808 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:09.621758    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" podStartSLOduration=7.621741267 podStartE2EDuration="7.621741267s" podCreationTimestamp="2026-05-26 12:18:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:18:09.620983109 +0000 UTC m=+3051.983932778" watchObservedRunningTime="2026-05-26 12:18:09.621741267 +0000 UTC m=+3051.984690938"
May 26 12:18:10.606405 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:10.606375    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:16.614751 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:16.614722    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:18:46.700656 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:46.700610    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 26 12:18:56.617877 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:18:56.617843    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:19:03.046509 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.046469    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"]
May 26 12:19:03.046959 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.046735    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kserve-container" containerID="cri-o://f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34" gracePeriod=30
May 26 12:19:03.046959 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.046832    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kube-rbac-proxy" containerID="cri-o://dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348" gracePeriod=30
May 26 12:19:03.126140 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126097    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"]
May 26 12:19:03.126467 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126451    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="storage-initializer"
May 26 12:19:03.126532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126471    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="storage-initializer"
May 26 12:19:03.126532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126499    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kserve-container"
May 26 12:19:03.126532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126506    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kserve-container"
May 26 12:19:03.126532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kube-rbac-proxy"
May 26 12:19:03.126532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126518    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kube-rbac-proxy"
May 26 12:19:03.126702 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126568    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kube-rbac-proxy"
May 26 12:19:03.126702 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.126576    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="83dd6908-8040-446a-b275-f567752f4b98" containerName="kserve-container"
May 26 12:19:03.129593 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.129574    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.131401 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.131381    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
May 26 12:19:03.131461 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.131382    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
May 26 12:19:03.139041 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.139012    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"]
May 26 12:19:03.210621 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.210580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7pw\" (UniqueName: \"kubernetes.io/projected/c8dd7415-cbce-4cc8-846d-98c19da62663-kube-api-access-gp7pw\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.210830 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.210643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8dd7415-cbce-4cc8-846d-98c19da62663-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.210830 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.210668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8dd7415-cbce-4cc8-846d-98c19da62663-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.210830 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.210705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8dd7415-cbce-4cc8-846d-98c19da62663-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.311151 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.311058    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8dd7415-cbce-4cc8-846d-98c19da62663-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.311151 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.311115    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7pw\" (UniqueName: \"kubernetes.io/projected/c8dd7415-cbce-4cc8-846d-98c19da62663-kube-api-access-gp7pw\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.311353 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.311161    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8dd7415-cbce-4cc8-846d-98c19da62663-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.311353 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.311181    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8dd7415-cbce-4cc8-846d-98c19da62663-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.311556 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.311534    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8dd7415-cbce-4cc8-846d-98c19da62663-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.311895 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.311858    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8dd7415-cbce-4cc8-846d-98c19da62663-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.313787 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.313766    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8dd7415-cbce-4cc8-846d-98c19da62663-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.320039 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.320001    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7pw\" (UniqueName: \"kubernetes.io/projected/c8dd7415-cbce-4cc8-846d-98c19da62663-kube-api-access-gp7pw\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-hp2dc\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.445071 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.445029    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:03.573039 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.572964    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"]
May 26 12:19:03.575866 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:19:03.575834    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8dd7415_cbce_4cc8_846d_98c19da62663.slice/crio-bf16feea4eb4fd329e5609fe236043e46eb15375ddd6cd74d63ee2eb6edadb4b WatchSource:0}: Error finding container bf16feea4eb4fd329e5609fe236043e46eb15375ddd6cd74d63ee2eb6edadb4b: Status 404 returned error can't find the container with id bf16feea4eb4fd329e5609fe236043e46eb15375ddd6cd74d63ee2eb6edadb4b
May 26 12:19:03.747640 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.747604    2563 generic.go:358] "Generic (PLEG): container finished" podID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerID="dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348" exitCode=2
May 26 12:19:03.747817 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.747673    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" event={"ID":"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba","Type":"ContainerDied","Data":"dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348"}
May 26 12:19:03.749065 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.749038    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" event={"ID":"c8dd7415-cbce-4cc8-846d-98c19da62663","Type":"ContainerStarted","Data":"a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151"}
May 26 12:19:03.749146 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:03.749078    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" event={"ID":"c8dd7415-cbce-4cc8-846d-98c19da62663","Type":"ContainerStarted","Data":"bf16feea4eb4fd329e5609fe236043e46eb15375ddd6cd74d63ee2eb6edadb4b"}
May 26 12:19:06.610105 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:06.610062    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
May 26 12:19:07.656936 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:07.656899    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 26 12:19:07.763047 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:07.763008    2563 generic.go:358] "Generic (PLEG): container finished" podID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerID="a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151" exitCode=0
May 26 12:19:07.763198 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:07.763076    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" event={"ID":"c8dd7415-cbce-4cc8-846d-98c19da62663","Type":"ContainerDied","Data":"a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151"}
May 26 12:19:08.767592 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:08.767555    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" event={"ID":"c8dd7415-cbce-4cc8-846d-98c19da62663","Type":"ContainerStarted","Data":"651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e"}
May 26 12:19:08.768103 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:08.767600    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" event={"ID":"c8dd7415-cbce-4cc8-846d-98c19da62663","Type":"ContainerStarted","Data":"b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e"}
May 26 12:19:08.768103 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:08.767817    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:08.787228 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:08.787186    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podStartSLOduration=5.787173153 podStartE2EDuration="5.787173153s" podCreationTimestamp="2026-05-26 12:19:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:19:08.786162811 +0000 UTC m=+3111.149112479" watchObservedRunningTime="2026-05-26 12:19:08.787173153 +0000 UTC m=+3111.150122822"
May 26 12:19:09.770419 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:09.770380    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:09.771618 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:09.771589    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:19:10.772911 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:10.772868    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:19:11.610004 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:11.609958    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
May 26 12:19:13.580109 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.580086    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:19:13.680865 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.680730    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g9tj\" (UniqueName: \"kubernetes.io/projected/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kube-api-access-9g9tj\") pod \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") "
May 26 12:19:13.680865 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.680816    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") "
May 26 12:19:13.680865 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.680846    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls\") pod \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") "
May 26 12:19:13.681114 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.680886    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kserve-provision-location\") pod \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\" (UID: \"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba\") "
May 26 12:19:13.681226 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.681198    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" (UID: "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:19:13.681305 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.681257    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" (UID: "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:19:13.683139 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.683118    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" (UID: "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:19:13.683259 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.683243    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kube-api-access-9g9tj" (OuterVolumeSpecName: "kube-api-access-9g9tj") pod "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" (UID: "ed11f405-8cfd-4ecd-82a8-2c32ce88eeba"). InnerVolumeSpecName "kube-api-access-9g9tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:19:13.782386 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.782348    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:19:13.782386 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.782379    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9tj\" (UniqueName: \"kubernetes.io/projected/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-kube-api-access-9g9tj\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:19:13.782592 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.782396    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:19:13.782592 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.782412    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:19:13.785656 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.785627    2563 generic.go:358] "Generic (PLEG): container finished" podID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerID="f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34" exitCode=0
May 26 12:19:13.785771 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.785673    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" event={"ID":"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba","Type":"ContainerDied","Data":"f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34"}
May 26 12:19:13.785771 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.785702    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq" event={"ID":"ed11f405-8cfd-4ecd-82a8-2c32ce88eeba","Type":"ContainerDied","Data":"439a6ba17b162070706b174c96cc72ef29520150736e16437c890516471c1c9e"}
May 26 12:19:13.785771 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.785725    2563 scope.go:117] "RemoveContainer" containerID="dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348"
May 26 12:19:13.785902 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.785725    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"
May 26 12:19:13.794182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.794163    2563 scope.go:117] "RemoveContainer" containerID="f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34"
May 26 12:19:13.801410 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.801389    2563 scope.go:117] "RemoveContainer" containerID="4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c"
May 26 12:19:13.809279 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.809255    2563 scope.go:117] "RemoveContainer" containerID="dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348"
May 26 12:19:13.809544 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:19:13.809522    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348\": container with ID starting with dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348 not found: ID does not exist" containerID="dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348"
May 26 12:19:13.809598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.809555    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348"} err="failed to get container status \"dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348\": rpc error: code = NotFound desc = could not find container \"dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348\": container with ID starting with dec3f34441b3aeaa9719ed966f8f625ff7bf77535f3410c17eb66d8887de9348 not found: ID does not exist"
May 26 12:19:13.809598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.809585    2563 scope.go:117] "RemoveContainer" containerID="f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34"
May 26 12:19:13.809748 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.809725    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"]
May 26 12:19:13.809853 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:19:13.809826    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34\": container with ID starting with f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34 not found: ID does not exist" containerID="f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34"
May 26 12:19:13.809942 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.809860    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34"} err="failed to get container status \"f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34\": rpc error: code = NotFound desc = could not find container \"f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34\": container with ID starting with f9ea07e0855533a20199c1f70aa12e15e6efad70663d947d7ecf25e688564e34 not found: ID does not exist"
May 26 12:19:13.809942 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.809878    2563 scope.go:117] "RemoveContainer" containerID="4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c"
May 26 12:19:13.810114 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:19:13.810099    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c\": container with ID starting with 4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c not found: ID does not exist" containerID="4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c"
May 26 12:19:13.810156 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.810116    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c"} err="failed to get container status \"4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c\": rpc error: code = NotFound desc = could not find container \"4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c\": container with ID starting with 4a625b723142bd234309e04e9a7d77cdf0733e3f79af0155ca58958f0290011c not found: ID does not exist"
May 26 12:19:13.813323 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:13.813301    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-99ldq"]
May 26 12:19:14.243024 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:14.242993    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" path="/var/lib/kubelet/pods/ed11f405-8cfd-4ecd-82a8-2c32ce88eeba/volumes"
May 26 12:19:15.777125 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:15.777097    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:19:15.777633 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:15.777606    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:19:25.777707 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:25.777662    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:19:35.778203 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:35.778122    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:19:45.778038 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:45.777998    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:19:55.777885 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:19:55.777842    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:20:05.778089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:05.778045    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:20:15.778716 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:15.778682    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:20:23.412711 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.412674    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"]
May 26 12:20:23.413185 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.413132    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" containerID="cri-o://b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e" gracePeriod=30
May 26 12:20:23.413250 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.413153    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kube-rbac-proxy" containerID="cri-o://651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e" gracePeriod=30
May 26 12:20:23.433811 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.433762    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"]
May 26 12:20:23.434042 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.434029    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="storage-initializer"
May 26 12:20:23.434089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.434044    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="storage-initializer"
May 26 12:20:23.434089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.434062    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kube-rbac-proxy"
May 26 12:20:23.434089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.434068    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kube-rbac-proxy"
May 26 12:20:23.434089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.434076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kserve-container"
May 26 12:20:23.434089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.434081    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kserve-container"
May 26 12:20:23.434254 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.434126    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kserve-container"
May 26 12:20:23.434254 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.434134    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed11f405-8cfd-4ecd-82a8-2c32ce88eeba" containerName="kube-rbac-proxy"
May 26 12:20:23.437115 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.437097    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.438948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.438920    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
May 26 12:20:23.439156 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.439139    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
May 26 12:20:23.450404 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.450378    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"]
May 26 12:20:23.501451 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.501416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291262d3-418b-4332-96cc-0a57a85704e5-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.501658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.501479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8664\" (UniqueName: \"kubernetes.io/projected/291262d3-418b-4332-96cc-0a57a85704e5-kube-api-access-w8664\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.501658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.501567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/291262d3-418b-4332-96cc-0a57a85704e5-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.501658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.501607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/291262d3-418b-4332-96cc-0a57a85704e5-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.602539 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.602483    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291262d3-418b-4332-96cc-0a57a85704e5-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.602539 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.602548    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w8664\" (UniqueName: \"kubernetes.io/projected/291262d3-418b-4332-96cc-0a57a85704e5-kube-api-access-w8664\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.602841 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.602575    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/291262d3-418b-4332-96cc-0a57a85704e5-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.602841 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.602596    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/291262d3-418b-4332-96cc-0a57a85704e5-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.602961 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.602930    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291262d3-418b-4332-96cc-0a57a85704e5-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.603277 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.603252    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/291262d3-418b-4332-96cc-0a57a85704e5-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.605236 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.605204    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/291262d3-418b-4332-96cc-0a57a85704e5-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.611607 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.611584    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8664\" (UniqueName: \"kubernetes.io/projected/291262d3-418b-4332-96cc-0a57a85704e5-kube-api-access-w8664\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.747476 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.747389    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:23.876810 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.876756    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"]
May 26 12:20:23.879654 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:20:23.879626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291262d3_418b_4332_96cc_0a57a85704e5.slice/crio-be01e100c5507974dbcd47d52999eccf1c6c3753983a84f648564654ecaeeb2c WatchSource:0}: Error finding container be01e100c5507974dbcd47d52999eccf1c6c3753983a84f648564654ecaeeb2c: Status 404 returned error can't find the container with id be01e100c5507974dbcd47d52999eccf1c6c3753983a84f648564654ecaeeb2c
May 26 12:20:23.881486 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.881469    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:20:23.971948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.971909    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" event={"ID":"291262d3-418b-4332-96cc-0a57a85704e5","Type":"ContainerStarted","Data":"d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8"}
May 26 12:20:23.971948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.971948    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" event={"ID":"291262d3-418b-4332-96cc-0a57a85704e5","Type":"ContainerStarted","Data":"be01e100c5507974dbcd47d52999eccf1c6c3753983a84f648564654ecaeeb2c"}
May 26 12:20:23.973638 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.973612    2563 generic.go:358] "Generic (PLEG): container finished" podID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerID="651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e" exitCode=2
May 26 12:20:23.973741 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:23.973654    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" event={"ID":"c8dd7415-cbce-4cc8-846d-98c19da62663","Type":"ContainerDied","Data":"651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e"}
May 26 12:20:25.774055 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:25.774012    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
May 26 12:20:25.778378 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:25.778349    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 12:20:27.860782 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.860755    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:20:27.935387 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.935348    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8dd7415-cbce-4cc8-846d-98c19da62663-proxy-tls\") pod \"c8dd7415-cbce-4cc8-846d-98c19da62663\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") "
May 26 12:20:27.935563 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.935399    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp7pw\" (UniqueName: \"kubernetes.io/projected/c8dd7415-cbce-4cc8-846d-98c19da62663-kube-api-access-gp7pw\") pod \"c8dd7415-cbce-4cc8-846d-98c19da62663\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") "
May 26 12:20:27.935563 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.935422    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8dd7415-cbce-4cc8-846d-98c19da62663-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"c8dd7415-cbce-4cc8-846d-98c19da62663\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") "
May 26 12:20:27.935563 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.935448    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8dd7415-cbce-4cc8-846d-98c19da62663-kserve-provision-location\") pod \"c8dd7415-cbce-4cc8-846d-98c19da62663\" (UID: \"c8dd7415-cbce-4cc8-846d-98c19da62663\") "
May 26 12:20:27.935870 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.935834    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dd7415-cbce-4cc8-846d-98c19da62663-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8dd7415-cbce-4cc8-846d-98c19da62663" (UID: "c8dd7415-cbce-4cc8-846d-98c19da62663"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:20:27.935870 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.935839    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8dd7415-cbce-4cc8-846d-98c19da62663-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "c8dd7415-cbce-4cc8-846d-98c19da62663" (UID: "c8dd7415-cbce-4cc8-846d-98c19da62663"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:20:27.937663 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.937643    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8dd7415-cbce-4cc8-846d-98c19da62663-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8dd7415-cbce-4cc8-846d-98c19da62663" (UID: "c8dd7415-cbce-4cc8-846d-98c19da62663"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:20:27.937910 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.937739    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dd7415-cbce-4cc8-846d-98c19da62663-kube-api-access-gp7pw" (OuterVolumeSpecName: "kube-api-access-gp7pw") pod "c8dd7415-cbce-4cc8-846d-98c19da62663" (UID: "c8dd7415-cbce-4cc8-846d-98c19da62663"). InnerVolumeSpecName "kube-api-access-gp7pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:20:27.985510 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.985423    2563 generic.go:358] "Generic (PLEG): container finished" podID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerID="b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e" exitCode=0
May 26 12:20:27.985663 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.985508    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"
May 26 12:20:27.985663 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.985509    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" event={"ID":"c8dd7415-cbce-4cc8-846d-98c19da62663","Type":"ContainerDied","Data":"b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e"}
May 26 12:20:27.985663 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.985554    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc" event={"ID":"c8dd7415-cbce-4cc8-846d-98c19da62663","Type":"ContainerDied","Data":"bf16feea4eb4fd329e5609fe236043e46eb15375ddd6cd74d63ee2eb6edadb4b"}
May 26 12:20:27.985663 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.985575    2563 scope.go:117] "RemoveContainer" containerID="651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e"
May 26 12:20:27.986895 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.986868    2563 generic.go:358] "Generic (PLEG): container finished" podID="291262d3-418b-4332-96cc-0a57a85704e5" containerID="d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8" exitCode=0
May 26 12:20:27.987009 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.986907    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" event={"ID":"291262d3-418b-4332-96cc-0a57a85704e5","Type":"ContainerDied","Data":"d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8"}
May 26 12:20:27.998338 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:27.998321    2563 scope.go:117] "RemoveContainer" containerID="b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e"
May 26 12:20:28.006067 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.006047    2563 scope.go:117] "RemoveContainer" containerID="a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151"
May 26 12:20:28.014931 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.014841    2563 scope.go:117] "RemoveContainer" containerID="651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e"
May 26 12:20:28.015167 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:20:28.015140    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e\": container with ID starting with 651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e not found: ID does not exist" containerID="651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e"
May 26 12:20:28.015262 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.015179    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e"} err="failed to get container status \"651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e\": rpc error: code = NotFound desc = could not find container \"651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e\": container with ID starting with 651cd854e2d985326ea9a0f5014012ea4900ccf37b3d8768d85a19fcdb8a075e not found: ID does not exist"
May 26 12:20:28.015262 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.015198    2563 scope.go:117] "RemoveContainer" containerID="b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e"
May 26 12:20:28.015483 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:20:28.015461    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e\": container with ID starting with b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e not found: ID does not exist" containerID="b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e"
May 26 12:20:28.015593 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.015488    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e"} err="failed to get container status \"b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e\": rpc error: code = NotFound desc = could not find container \"b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e\": container with ID starting with b2f805ae77a392e9bee225e596ee20b9d0d40156d6243d7118a5ee61a89cf44e not found: ID does not exist"
May 26 12:20:28.015593 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.015507    2563 scope.go:117] "RemoveContainer" containerID="a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151"
May 26 12:20:28.015830 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:20:28.015811    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151\": container with ID starting with a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151 not found: ID does not exist" containerID="a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151"
May 26 12:20:28.015900 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.015837    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151"} err="failed to get container status \"a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151\": rpc error: code = NotFound desc = could not find container \"a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151\": container with ID starting with a2121747a516adc6a762b926611866a890953aa85cae35e9d7db6d5bd3c2a151 not found: ID does not exist"
May 26 12:20:28.024555 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.024532    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"]
May 26 12:20:28.031750 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.031724    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-hp2dc"]
May 26 12:20:28.036606 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.036584    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8dd7415-cbce-4cc8-846d-98c19da62663-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:20:28.036717 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.036677    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7pw\" (UniqueName: \"kubernetes.io/projected/c8dd7415-cbce-4cc8-846d-98c19da62663-kube-api-access-gp7pw\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:20:28.036837 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.036822    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8dd7415-cbce-4cc8-846d-98c19da62663-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:20:28.036910 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.036842    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8dd7415-cbce-4cc8-846d-98c19da62663-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:20:28.242603 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.242506    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" path="/var/lib/kubelet/pods/c8dd7415-cbce-4cc8-846d-98c19da62663/volumes"
May 26 12:20:28.992471 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.992432    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" event={"ID":"291262d3-418b-4332-96cc-0a57a85704e5","Type":"ContainerStarted","Data":"2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa"}
May 26 12:20:28.992866 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.992477    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" event={"ID":"291262d3-418b-4332-96cc-0a57a85704e5","Type":"ContainerStarted","Data":"b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c"}
May 26 12:20:28.992866 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:28.992686    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:29.011451 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:29.011407    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podStartSLOduration=6.011392674 podStartE2EDuration="6.011392674s" podCreationTimestamp="2026-05-26 12:20:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:20:29.010175339 +0000 UTC m=+3191.373125009" watchObservedRunningTime="2026-05-26 12:20:29.011392674 +0000 UTC m=+3191.374342343"
May 26 12:20:29.995540 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:29.995500    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:29.996905 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:29.996877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:20:30.997667 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:30.997628    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:20:36.004068 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:36.004025    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:20:36.004636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:36.004608    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:20:46.005465 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:46.005418    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:20:56.005414 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:20:56.005372    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:21:06.005372 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:06.005288    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:21:16.005264 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:16.005227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:21:26.005446 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:26.005405    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:21:36.005942 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:36.005911    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:21:43.521838 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.521777    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"]
May 26 12:21:43.522300 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.522138    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" containerID="cri-o://b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c" gracePeriod=30
May 26 12:21:43.522300 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.522166    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kube-rbac-proxy" containerID="cri-o://2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa" gracePeriod=30
May 26 12:21:43.615022 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.614985    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"]
May 26 12:21:43.615302 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.615289    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container"
May 26 12:21:43.615345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.615305    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container"
May 26 12:21:43.615345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.615318    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="storage-initializer"
May 26 12:21:43.615345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.615323    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="storage-initializer"
May 26 12:21:43.615345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.615334    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kube-rbac-proxy"
May 26 12:21:43.615345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.615340    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kube-rbac-proxy"
May 26 12:21:43.615498 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.615380    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kube-rbac-proxy"
May 26 12:21:43.615498 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.615390    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="c8dd7415-cbce-4cc8-846d-98c19da62663" containerName="kserve-container"
May 26 12:21:43.618485 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.618469    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.620360 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.620339    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
May 26 12:21:43.620522 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.620507    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
May 26 12:21:43.630543 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.630521    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"]
May 26 12:21:43.776609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.776513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f985c3a-9c52-4206-833f-9629e16fd545-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.776609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.776576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f985c3a-9c52-4206-833f-9629e16fd545-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.776609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.776601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f985c3a-9c52-4206-833f-9629e16fd545-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.776888 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.776631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9xc\" (UniqueName: \"kubernetes.io/projected/8f985c3a-9c52-4206-833f-9629e16fd545-kube-api-access-kq9xc\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.877395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.877361    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f985c3a-9c52-4206-833f-9629e16fd545-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.877395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.877402    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f985c3a-9c52-4206-833f-9629e16fd545-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.877622 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.877429    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9xc\" (UniqueName: \"kubernetes.io/projected/8f985c3a-9c52-4206-833f-9629e16fd545-kube-api-access-kq9xc\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.877622 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.877466    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f985c3a-9c52-4206-833f-9629e16fd545-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.877830 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.877780    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f985c3a-9c52-4206-833f-9629e16fd545-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.878178 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.878159    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f985c3a-9c52-4206-833f-9629e16fd545-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.880170 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.880147    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f985c3a-9c52-4206-833f-9629e16fd545-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.885969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.885948    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9xc\" (UniqueName: \"kubernetes.io/projected/8f985c3a-9c52-4206-833f-9629e16fd545-kube-api-access-kq9xc\") pod \"isvc-tensorflow-predictor-6756f669d7-74h6x\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:43.927958 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:43.927917    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:44.054095 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:44.054064    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"]
May 26 12:21:44.057377 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:21:44.057350    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f985c3a_9c52_4206_833f_9629e16fd545.slice/crio-4f48f68c35e8474ab8b11f1e6ace605e94fdb6ec0125063edcc65db8e49bf775 WatchSource:0}: Error finding container 4f48f68c35e8474ab8b11f1e6ace605e94fdb6ec0125063edcc65db8e49bf775: Status 404 returned error can't find the container with id 4f48f68c35e8474ab8b11f1e6ace605e94fdb6ec0125063edcc65db8e49bf775
May 26 12:21:44.195642 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:44.195611    2563 generic.go:358] "Generic (PLEG): container finished" podID="291262d3-418b-4332-96cc-0a57a85704e5" containerID="2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa" exitCode=2
May 26 12:21:44.195849 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:44.195679    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" event={"ID":"291262d3-418b-4332-96cc-0a57a85704e5","Type":"ContainerDied","Data":"2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa"}
May 26 12:21:44.197060 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:44.197034    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" event={"ID":"8f985c3a-9c52-4206-833f-9629e16fd545","Type":"ContainerStarted","Data":"8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23"}
May 26 12:21:44.197169 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:44.197067    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" event={"ID":"8f985c3a-9c52-4206-833f-9629e16fd545","Type":"ContainerStarted","Data":"4f48f68c35e8474ab8b11f1e6ace605e94fdb6ec0125063edcc65db8e49bf775"}
May 26 12:21:45.998711 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:45.998674    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
May 26 12:21:46.005188 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:46.005159    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 12:21:47.862400 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:47.862376    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:21:48.007720 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.007633    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291262d3-418b-4332-96cc-0a57a85704e5-kserve-provision-location\") pod \"291262d3-418b-4332-96cc-0a57a85704e5\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") "
May 26 12:21:48.007720 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.007674    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8664\" (UniqueName: \"kubernetes.io/projected/291262d3-418b-4332-96cc-0a57a85704e5-kube-api-access-w8664\") pod \"291262d3-418b-4332-96cc-0a57a85704e5\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") "
May 26 12:21:48.007720 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.007704    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/291262d3-418b-4332-96cc-0a57a85704e5-proxy-tls\") pod \"291262d3-418b-4332-96cc-0a57a85704e5\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") "
May 26 12:21:48.008042 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.007824    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/291262d3-418b-4332-96cc-0a57a85704e5-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"291262d3-418b-4332-96cc-0a57a85704e5\" (UID: \"291262d3-418b-4332-96cc-0a57a85704e5\") "
May 26 12:21:48.008042 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.008028    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291262d3-418b-4332-96cc-0a57a85704e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "291262d3-418b-4332-96cc-0a57a85704e5" (UID: "291262d3-418b-4332-96cc-0a57a85704e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:21:48.008267 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.008238    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291262d3-418b-4332-96cc-0a57a85704e5-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "291262d3-418b-4332-96cc-0a57a85704e5" (UID: "291262d3-418b-4332-96cc-0a57a85704e5"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:21:48.010010 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.009981    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/291262d3-418b-4332-96cc-0a57a85704e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "291262d3-418b-4332-96cc-0a57a85704e5" (UID: "291262d3-418b-4332-96cc-0a57a85704e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:21:48.010010 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.009985    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291262d3-418b-4332-96cc-0a57a85704e5-kube-api-access-w8664" (OuterVolumeSpecName: "kube-api-access-w8664") pod "291262d3-418b-4332-96cc-0a57a85704e5" (UID: "291262d3-418b-4332-96cc-0a57a85704e5"). InnerVolumeSpecName "kube-api-access-w8664". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:21:48.108427 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.108392    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/291262d3-418b-4332-96cc-0a57a85704e5-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:21:48.108427 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.108422    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/291262d3-418b-4332-96cc-0a57a85704e5-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:21:48.108427 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.108433    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8664\" (UniqueName: \"kubernetes.io/projected/291262d3-418b-4332-96cc-0a57a85704e5-kube-api-access-w8664\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:21:48.108647 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.108443    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/291262d3-418b-4332-96cc-0a57a85704e5-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:21:48.210710 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.210680    2563 generic.go:358] "Generic (PLEG): container finished" podID="291262d3-418b-4332-96cc-0a57a85704e5" containerID="b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c" exitCode=0
May 26 12:21:48.210892 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.210745    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" event={"ID":"291262d3-418b-4332-96cc-0a57a85704e5","Type":"ContainerDied","Data":"b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c"}
May 26 12:21:48.210892 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.210752    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"
May 26 12:21:48.210892 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.210781    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr" event={"ID":"291262d3-418b-4332-96cc-0a57a85704e5","Type":"ContainerDied","Data":"be01e100c5507974dbcd47d52999eccf1c6c3753983a84f648564654ecaeeb2c"}
May 26 12:21:48.210892 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.210827    2563 scope.go:117] "RemoveContainer" containerID="2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa"
May 26 12:21:48.222409 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.222388    2563 scope.go:117] "RemoveContainer" containerID="b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c"
May 26 12:21:48.232705 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.232687    2563 scope.go:117] "RemoveContainer" containerID="d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8"
May 26 12:21:48.237865 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.237834    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"]
May 26 12:21:48.239673 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.239656    2563 scope.go:117] "RemoveContainer" containerID="2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa"
May 26 12:21:48.239969 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:21:48.239944    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa\": container with ID starting with 2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa not found: ID does not exist" containerID="2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa"
May 26 12:21:48.240043 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.239985    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa"} err="failed to get container status \"2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa\": rpc error: code = NotFound desc = could not find container \"2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa\": container with ID starting with 2bf2fed9665ddaf343c16f1eec1a002ff0eefb9049ebe8ed6dbfe94e72dcd4aa not found: ID does not exist"
May 26 12:21:48.240043 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.240002    2563 scope.go:117] "RemoveContainer" containerID="b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c"
May 26 12:21:48.240273 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:21:48.240251    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c\": container with ID starting with b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c not found: ID does not exist" containerID="b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c"
May 26 12:21:48.240335 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.240284    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c"} err="failed to get container status \"b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c\": rpc error: code = NotFound desc = could not find container \"b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c\": container with ID starting with b6175717a8ef6c1bd7bc96bb11cb0523cfadd7f8f23617ef44797c8f4ea1ac8c not found: ID does not exist"
May 26 12:21:48.240393 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.240338    2563 scope.go:117] "RemoveContainer" containerID="d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8"
May 26 12:21:48.241028 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:21:48.240991    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8\": container with ID starting with d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8 not found: ID does not exist" containerID="d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8"
May 26 12:21:48.241028 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.241021    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8"} err="failed to get container status \"d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8\": rpc error: code = NotFound desc = could not find container \"d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8\": container with ID starting with d7038b93917df50fc662d54ef79f8dd6a12234c10d6849eb6b33d57b33c8a7a8 not found: ID does not exist"
May 26 12:21:48.243324 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:48.243307    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-wq7sr"]
May 26 12:21:49.215717 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:49.215686    2563 generic.go:358] "Generic (PLEG): container finished" podID="8f985c3a-9c52-4206-833f-9629e16fd545" containerID="8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23" exitCode=0
May 26 12:21:49.216111 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:49.215751    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" event={"ID":"8f985c3a-9c52-4206-833f-9629e16fd545","Type":"ContainerDied","Data":"8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23"}
May 26 12:21:50.243787 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:50.243747    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291262d3-418b-4332-96cc-0a57a85704e5" path="/var/lib/kubelet/pods/291262d3-418b-4332-96cc-0a57a85704e5/volumes"
May 26 12:21:53.232016 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:53.231973    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" event={"ID":"8f985c3a-9c52-4206-833f-9629e16fd545","Type":"ContainerStarted","Data":"1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3"}
May 26 12:21:53.232016 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:53.232024    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" event={"ID":"8f985c3a-9c52-4206-833f-9629e16fd545","Type":"ContainerStarted","Data":"559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622"}
May 26 12:21:53.232522 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:53.232249    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:53.250457 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:53.250415    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podStartSLOduration=6.393237189 podStartE2EDuration="10.250402682s" podCreationTimestamp="2026-05-26 12:21:43 +0000 UTC" firstStartedPulling="2026-05-26 12:21:49.216980102 +0000 UTC m=+3271.579929748" lastFinishedPulling="2026-05-26 12:21:53.074145593 +0000 UTC m=+3275.437095241" observedRunningTime="2026-05-26 12:21:53.248763915 +0000 UTC m=+3275.611713584" watchObservedRunningTime="2026-05-26 12:21:53.250402682 +0000 UTC m=+3275.613352349"
May 26 12:21:54.235238 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:54.235205    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:21:54.236314 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:54.236287    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 26 12:21:55.237924 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:21:55.237866    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 26 12:22:00.242521 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:00.242488    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:22:00.243089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:00.243059    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 26 12:22:10.243928 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:10.243900    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:22:24.278372 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278337    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"]
May 26 12:22:24.278746 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container"
May 26 12:22:24.278746 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container"
May 26 12:22:24.278746 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278593    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="storage-initializer"
May 26 12:22:24.278746 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278599    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="storage-initializer"
May 26 12:22:24.278746 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278617    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kube-rbac-proxy"
May 26 12:22:24.278746 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278623    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kube-rbac-proxy"
May 26 12:22:24.278746 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278662    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kserve-container"
May 26 12:22:24.278746 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.278670    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="291262d3-418b-4332-96cc-0a57a85704e5" containerName="kube-rbac-proxy"
May 26 12:22:24.283604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.283579    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.296361 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.296341    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
May 26 12:22:24.296852 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.296828    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
May 26 12:22:24.314146 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.314124    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"]
May 26 12:22:24.376578 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.376536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htbdd\" (UniqueName: \"kubernetes.io/projected/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kube-api-access-htbdd\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.376754 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.376583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9c3fe54-1747-41c3-ae38-be4bc280b15c-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.376754 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.376643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.376754 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.376706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9c3fe54-1747-41c3-ae38-be4bc280b15c-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.421260 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.421223    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"]
May 26 12:22:24.421643 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.421596    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kserve-container" containerID="cri-o://559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622" gracePeriod=30
May 26 12:22:24.421736 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.421646    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy" containerID="cri-o://1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3" gracePeriod=30
May 26 12:22:24.477993 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.477953    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-htbdd\" (UniqueName: \"kubernetes.io/projected/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kube-api-access-htbdd\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.477993 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.477993    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9c3fe54-1747-41c3-ae38-be4bc280b15c-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.478257 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.478012    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.478257 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.478053    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9c3fe54-1747-41c3-ae38-be4bc280b15c-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.478504 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.478481    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.478736 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.478719    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9c3fe54-1747-41c3-ae38-be4bc280b15c-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.480585 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.480565    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9c3fe54-1747-41c3-ae38-be4bc280b15c-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.485671 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.485646    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htbdd\" (UniqueName: \"kubernetes.io/projected/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kube-api-access-htbdd\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.593535 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.593494    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:24.718062 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:24.718031    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"]
May 26 12:22:24.721144 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:22:24.721120    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c3fe54_1747_41c3_ae38_be4bc280b15c.slice/crio-503e4dce5ae54df1d38c66e3180fed2295a135f8852346665549c75157c1f881 WatchSource:0}: Error finding container 503e4dce5ae54df1d38c66e3180fed2295a135f8852346665549c75157c1f881: Status 404 returned error can't find the container with id 503e4dce5ae54df1d38c66e3180fed2295a135f8852346665549c75157c1f881
May 26 12:22:25.238600 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:25.238560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 26 12:22:25.320299 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:25.320267    2563 generic.go:358] "Generic (PLEG): container finished" podID="8f985c3a-9c52-4206-833f-9629e16fd545" containerID="1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3" exitCode=2
May 26 12:22:25.320734 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:25.320338    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" event={"ID":"8f985c3a-9c52-4206-833f-9629e16fd545","Type":"ContainerDied","Data":"1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3"}
May 26 12:22:25.321761 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:25.321737    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" event={"ID":"e9c3fe54-1747-41c3-ae38-be4bc280b15c","Type":"ContainerStarted","Data":"c9ec5e3e95453b579e36453931a8af8846f5e7548f15408c76abb6582c00024a"}
May 26 12:22:25.321884 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:25.321769    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" event={"ID":"e9c3fe54-1747-41c3-ae38-be4bc280b15c","Type":"ContainerStarted","Data":"503e4dce5ae54df1d38c66e3180fed2295a135f8852346665549c75157c1f881"}
May 26 12:22:29.338008 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:29.336303    2563 generic.go:358] "Generic (PLEG): container finished" podID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerID="c9ec5e3e95453b579e36453931a8af8846f5e7548f15408c76abb6582c00024a" exitCode=0
May 26 12:22:29.338008 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:29.336408    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" event={"ID":"e9c3fe54-1747-41c3-ae38-be4bc280b15c","Type":"ContainerDied","Data":"c9ec5e3e95453b579e36453931a8af8846f5e7548f15408c76abb6582c00024a"}
May 26 12:22:30.239057 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:30.239007    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 26 12:22:30.340991 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:30.340958    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" event={"ID":"e9c3fe54-1747-41c3-ae38-be4bc280b15c","Type":"ContainerStarted","Data":"96d0f575567f967035eda815c8c1d3a7946cfa4d9c60304cb44d7cac1f07634e"}
May 26 12:22:30.340991 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:30.340991    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" event={"ID":"e9c3fe54-1747-41c3-ae38-be4bc280b15c","Type":"ContainerStarted","Data":"4f9bc7c80fa870840ae513b802d33dc6646cd6bc58b13ed0ed20087b92a1e3ed"}
May 26 12:22:30.341531 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:30.341326    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:30.341531 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:30.341444    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:30.342828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:30.342782    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 26 12:22:30.359061 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:30.358426    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podStartSLOduration=6.358411316 podStartE2EDuration="6.358411316s" podCreationTimestamp="2026-05-26 12:22:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:22:30.357120876 +0000 UTC m=+3312.720070545" watchObservedRunningTime="2026-05-26 12:22:30.358411316 +0000 UTC m=+3312.721360985"
May 26 12:22:31.344266 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:31.344220    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 26 12:22:32.346389 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:32.346347    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 26 12:22:35.238378 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:35.238332    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 26 12:22:35.238763 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:35.238448    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:22:37.350600 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:37.350517    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:37.351138 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:37.351109    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 26 12:22:40.238276 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:40.238226    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 26 12:22:45.238076 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:45.238023    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 26 12:22:47.352212 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:47.352183    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:22:50.238874 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:50.238826    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 26 12:22:55.067418 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.067394    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:22:55.201992 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.201896    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f985c3a-9c52-4206-833f-9629e16fd545-kserve-provision-location\") pod \"8f985c3a-9c52-4206-833f-9629e16fd545\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") "
May 26 12:22:55.201992 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.201964    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f985c3a-9c52-4206-833f-9629e16fd545-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"8f985c3a-9c52-4206-833f-9629e16fd545\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") "
May 26 12:22:55.201992 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.201986    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq9xc\" (UniqueName: \"kubernetes.io/projected/8f985c3a-9c52-4206-833f-9629e16fd545-kube-api-access-kq9xc\") pod \"8f985c3a-9c52-4206-833f-9629e16fd545\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") "
May 26 12:22:55.202241 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.202025    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f985c3a-9c52-4206-833f-9629e16fd545-proxy-tls\") pod \"8f985c3a-9c52-4206-833f-9629e16fd545\" (UID: \"8f985c3a-9c52-4206-833f-9629e16fd545\") "
May 26 12:22:55.202409 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.202372    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f985c3a-9c52-4206-833f-9629e16fd545-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "8f985c3a-9c52-4206-833f-9629e16fd545" (UID: "8f985c3a-9c52-4206-833f-9629e16fd545"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:22:55.204251 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.204223    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f985c3a-9c52-4206-833f-9629e16fd545-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f985c3a-9c52-4206-833f-9629e16fd545" (UID: "8f985c3a-9c52-4206-833f-9629e16fd545"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:22:55.204337 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.204308    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f985c3a-9c52-4206-833f-9629e16fd545-kube-api-access-kq9xc" (OuterVolumeSpecName: "kube-api-access-kq9xc") pod "8f985c3a-9c52-4206-833f-9629e16fd545" (UID: "8f985c3a-9c52-4206-833f-9629e16fd545"). InnerVolumeSpecName "kube-api-access-kq9xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:22:55.211110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.211085    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f985c3a-9c52-4206-833f-9629e16fd545-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f985c3a-9c52-4206-833f-9629e16fd545" (UID: "8f985c3a-9c52-4206-833f-9629e16fd545"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:22:55.302476 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.302440    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f985c3a-9c52-4206-833f-9629e16fd545-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:22:55.302476 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.302469    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq9xc\" (UniqueName: \"kubernetes.io/projected/8f985c3a-9c52-4206-833f-9629e16fd545-kube-api-access-kq9xc\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:22:55.302476 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.302480    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f985c3a-9c52-4206-833f-9629e16fd545-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:22:55.302701 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.302489    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f985c3a-9c52-4206-833f-9629e16fd545-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:22:55.408303 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.408266    2563 generic.go:358] "Generic (PLEG): container finished" podID="8f985c3a-9c52-4206-833f-9629e16fd545" containerID="559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622" exitCode=137
May 26 12:22:55.408474 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.408350    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" event={"ID":"8f985c3a-9c52-4206-833f-9629e16fd545","Type":"ContainerDied","Data":"559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622"}
May 26 12:22:55.408474 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.408401    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x" event={"ID":"8f985c3a-9c52-4206-833f-9629e16fd545","Type":"ContainerDied","Data":"4f48f68c35e8474ab8b11f1e6ace605e94fdb6ec0125063edcc65db8e49bf775"}
May 26 12:22:55.408474 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.408418    2563 scope.go:117] "RemoveContainer" containerID="1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3"
May 26 12:22:55.408474 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.408362    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"
May 26 12:22:55.416316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.416115    2563 scope.go:117] "RemoveContainer" containerID="559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622"
May 26 12:22:55.422954 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.422939    2563 scope.go:117] "RemoveContainer" containerID="8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23"
May 26 12:22:55.430099 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.429895    2563 scope.go:117] "RemoveContainer" containerID="1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3"
May 26 12:22:55.430214 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:22:55.430195    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3\": container with ID starting with 1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3 not found: ID does not exist" containerID="1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3"
May 26 12:22:55.430281 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.430230    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3"} err="failed to get container status \"1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3\": rpc error: code = NotFound desc = could not find container \"1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3\": container with ID starting with 1949f83a18f4c15f5ebd7a97c59a4e668000b886d72aadaa04aa6e66d3a7c5f3 not found: ID does not exist"
May 26 12:22:55.430281 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.430260    2563 scope.go:117] "RemoveContainer" containerID="559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622"
May 26 12:22:55.430571 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:22:55.430547    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622\": container with ID starting with 559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622 not found: ID does not exist" containerID="559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622"
May 26 12:22:55.430654 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.430580    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622"} err="failed to get container status \"559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622\": rpc error: code = NotFound desc = could not find container \"559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622\": container with ID starting with 559cb765ab77c3c6fe05c50b5998e59f771711f6f2aa0bfe69b9a85a87da5622 not found: ID does not exist"
May 26 12:22:55.430654 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.430604    2563 scope.go:117] "RemoveContainer" containerID="8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23"
May 26 12:22:55.430747 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.430680    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"]
May 26 12:22:55.430904 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:22:55.430883    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23\": container with ID starting with 8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23 not found: ID does not exist" containerID="8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23"
May 26 12:22:55.430963 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.430911    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23"} err="failed to get container status \"8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23\": rpc error: code = NotFound desc = could not find container \"8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23\": container with ID starting with 8a1f165e3284c28c42324becf4fa5343cf8d3dbaa72310d205768ca96c251d23 not found: ID does not exist"
May 26 12:22:55.437181 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:55.437162    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-74h6x"]
May 26 12:22:56.242032 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:22:56.241990    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" path="/var/lib/kubelet/pods/8f985c3a-9c52-4206-833f-9629e16fd545/volumes"
May 26 12:23:01.459426 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:01.459399    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:23:01.464173 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:01.464150    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:23:05.257244 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.257210    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"]
May 26 12:23:05.257779 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.257524    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kserve-container" containerID="cri-o://4f9bc7c80fa870840ae513b802d33dc6646cd6bc58b13ed0ed20087b92a1e3ed" gracePeriod=30
May 26 12:23:05.257779 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.257586    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy" containerID="cri-o://96d0f575567f967035eda815c8c1d3a7946cfa4d9c60304cb44d7cac1f07634e" gracePeriod=30
May 26 12:23:05.318535 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318498    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"]
May 26 12:23:05.318751 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318739    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="storage-initializer"
May 26 12:23:05.318819 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318752    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="storage-initializer"
May 26 12:23:05.318819 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318807    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy"
May 26 12:23:05.318819 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318817    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy"
May 26 12:23:05.318919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318834    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kserve-container"
May 26 12:23:05.318919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318841    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kserve-container"
May 26 12:23:05.318919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318886    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kube-rbac-proxy"
May 26 12:23:05.318919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.318894    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f985c3a-9c52-4206-833f-9629e16fd545" containerName="kserve-container"
May 26 12:23:05.321815 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.321774    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.323563 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.323544    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
May 26 12:23:05.323666 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.323646    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
May 26 12:23:05.332040 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.332011    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"]
May 26 12:23:05.442332 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.442301    2563 generic.go:358] "Generic (PLEG): container finished" podID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerID="96d0f575567f967035eda815c8c1d3a7946cfa4d9c60304cb44d7cac1f07634e" exitCode=2
May 26 12:23:05.442512 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.442359    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" event={"ID":"e9c3fe54-1747-41c3-ae38-be4bc280b15c","Type":"ContainerDied","Data":"96d0f575567f967035eda815c8c1d3a7946cfa4d9c60304cb44d7cac1f07634e"}
May 26 12:23:05.474700 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.474661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b19391-e8b8-4d04-ae77-63fc264740dc-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.474879 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.474732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b19391-e8b8-4d04-ae77-63fc264740dc-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.474879 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.474830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgwls\" (UniqueName: \"kubernetes.io/projected/24b19391-e8b8-4d04-ae77-63fc264740dc-kube-api-access-vgwls\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.474879 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.474869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b19391-e8b8-4d04-ae77-63fc264740dc-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.575902 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.575868    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b19391-e8b8-4d04-ae77-63fc264740dc-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.576055 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.575918    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vgwls\" (UniqueName: \"kubernetes.io/projected/24b19391-e8b8-4d04-ae77-63fc264740dc-kube-api-access-vgwls\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.576055 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.575939    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b19391-e8b8-4d04-ae77-63fc264740dc-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.576055 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.575972    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b19391-e8b8-4d04-ae77-63fc264740dc-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.576337 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.576315    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b19391-e8b8-4d04-ae77-63fc264740dc-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.576609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.576588    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b19391-e8b8-4d04-ae77-63fc264740dc-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.578494 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.578473    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b19391-e8b8-4d04-ae77-63fc264740dc-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.583495 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.583474    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgwls\" (UniqueName: \"kubernetes.io/projected/24b19391-e8b8-4d04-ae77-63fc264740dc-kube-api-access-vgwls\") pod \"isvc-triton-predictor-84bb65d94b-tf6gl\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.632478 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.632437    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:23:05.760665 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:05.760630    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"]
May 26 12:23:05.764339 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:23:05.764313    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b19391_e8b8_4d04_ae77_63fc264740dc.slice/crio-99288c06b2cffe3fdb85948f6bb9d8b55c2b86454a68a73b4f408064009c9b6f WatchSource:0}: Error finding container 99288c06b2cffe3fdb85948f6bb9d8b55c2b86454a68a73b4f408064009c9b6f: Status 404 returned error can't find the container with id 99288c06b2cffe3fdb85948f6bb9d8b55c2b86454a68a73b4f408064009c9b6f
May 26 12:23:06.446703 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:06.446667    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" event={"ID":"24b19391-e8b8-4d04-ae77-63fc264740dc","Type":"ContainerStarted","Data":"451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215"}
May 26 12:23:06.446703 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:06.446703    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" event={"ID":"24b19391-e8b8-4d04-ae77-63fc264740dc","Type":"ContainerStarted","Data":"99288c06b2cffe3fdb85948f6bb9d8b55c2b86454a68a73b4f408064009c9b6f"}
May 26 12:23:07.346788 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:07.346746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 26 12:23:10.458373 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:10.458338    2563 generic.go:358] "Generic (PLEG): container finished" podID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerID="451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215" exitCode=0
May 26 12:23:10.458373 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:10.458378    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" event={"ID":"24b19391-e8b8-4d04-ae77-63fc264740dc","Type":"ContainerDied","Data":"451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215"}
May 26 12:23:12.347516 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:12.346641    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 26 12:23:17.347943 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:17.347483    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 26 12:23:17.347943 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:17.347629    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:23:22.346811 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:22.346749    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 26 12:23:27.346711 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:27.346657    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 26 12:23:32.347625 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:32.347570    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 26 12:23:35.291435 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:23:35.291396    2563 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c3fe54_1747_41c3_ae38_be4bc280b15c.slice/crio-503e4dce5ae54df1d38c66e3180fed2295a135f8852346665549c75157c1f881\": RecentStats: unable to find data in memory cache]"
May 26 12:23:35.558783 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:35.558739    2563 generic.go:358] "Generic (PLEG): container finished" podID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerID="4f9bc7c80fa870840ae513b802d33dc6646cd6bc58b13ed0ed20087b92a1e3ed" exitCode=137
May 26 12:23:35.558986 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:35.558829    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" event={"ID":"e9c3fe54-1747-41c3-ae38-be4bc280b15c","Type":"ContainerDied","Data":"4f9bc7c80fa870840ae513b802d33dc6646cd6bc58b13ed0ed20087b92a1e3ed"}
May 26 12:23:35.946982 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:35.946952    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:23:36.032257 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.032215    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kserve-provision-location\") pod \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") "
May 26 12:23:36.032480 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.032268    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htbdd\" (UniqueName: \"kubernetes.io/projected/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kube-api-access-htbdd\") pod \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") "
May 26 12:23:36.032480 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.032304    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9c3fe54-1747-41c3-ae38-be4bc280b15c-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") "
May 26 12:23:36.032480 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.032330    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9c3fe54-1747-41c3-ae38-be4bc280b15c-proxy-tls\") pod \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\" (UID: \"e9c3fe54-1747-41c3-ae38-be4bc280b15c\") "
May 26 12:23:36.032750 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.032716    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c3fe54-1747-41c3-ae38-be4bc280b15c-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "e9c3fe54-1747-41c3-ae38-be4bc280b15c" (UID: "e9c3fe54-1747-41c3-ae38-be4bc280b15c"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:23:36.035965 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.035716    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c3fe54-1747-41c3-ae38-be4bc280b15c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9c3fe54-1747-41c3-ae38-be4bc280b15c" (UID: "e9c3fe54-1747-41c3-ae38-be4bc280b15c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:23:36.036558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.036532    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kube-api-access-htbdd" (OuterVolumeSpecName: "kube-api-access-htbdd") pod "e9c3fe54-1747-41c3-ae38-be4bc280b15c" (UID: "e9c3fe54-1747-41c3-ae38-be4bc280b15c"). InnerVolumeSpecName "kube-api-access-htbdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:23:36.041195 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.040922    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9c3fe54-1747-41c3-ae38-be4bc280b15c" (UID: "e9c3fe54-1747-41c3-ae38-be4bc280b15c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:23:36.133680 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.133593    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htbdd\" (UniqueName: \"kubernetes.io/projected/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kube-api-access-htbdd\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:23:36.133680 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.133629    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9c3fe54-1747-41c3-ae38-be4bc280b15c-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:23:36.133680 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.133640    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9c3fe54-1747-41c3-ae38-be4bc280b15c-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:23:36.133680 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.133649    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9c3fe54-1747-41c3-ae38-be4bc280b15c-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:23:36.564774 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.564732    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5" event={"ID":"e9c3fe54-1747-41c3-ae38-be4bc280b15c","Type":"ContainerDied","Data":"503e4dce5ae54df1d38c66e3180fed2295a135f8852346665549c75157c1f881"}
May 26 12:23:36.565260 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.564787    2563 scope.go:117] "RemoveContainer" containerID="96d0f575567f967035eda815c8c1d3a7946cfa4d9c60304cb44d7cac1f07634e"
May 26 12:23:36.565260 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.564839    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"
May 26 12:23:36.575240 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.575221    2563 scope.go:117] "RemoveContainer" containerID="4f9bc7c80fa870840ae513b802d33dc6646cd6bc58b13ed0ed20087b92a1e3ed"
May 26 12:23:36.582656 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.582612    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"]
May 26 12:23:36.586877 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.586827    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-qzfw5"]
May 26 12:23:36.588463 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:36.588386    2563 scope.go:117] "RemoveContainer" containerID="c9ec5e3e95453b579e36453931a8af8846f5e7548f15408c76abb6582c00024a"
May 26 12:23:38.243658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:23:38.243618    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" path="/var/lib/kubelet/pods/e9c3fe54-1747-41c3-ae38-be4bc280b15c/volumes"
May 26 12:25:04.823306 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:04.823271    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" event={"ID":"24b19391-e8b8-4d04-ae77-63fc264740dc","Type":"ContainerStarted","Data":"e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6"}
May 26 12:25:04.823306 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:04.823305    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" event={"ID":"24b19391-e8b8-4d04-ae77-63fc264740dc","Type":"ContainerStarted","Data":"c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b"}
May 26 12:25:04.823742 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:04.823403    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:25:04.845030 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:04.844948    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" podStartSLOduration=5.721374595 podStartE2EDuration="1m59.844929662s" podCreationTimestamp="2026-05-26 12:23:05 +0000 UTC" firstStartedPulling="2026-05-26 12:23:10.45941491 +0000 UTC m=+3352.822364557" lastFinishedPulling="2026-05-26 12:25:04.582969977 +0000 UTC m=+3466.945919624" observedRunningTime="2026-05-26 12:25:04.843733305 +0000 UTC m=+3467.206682974" watchObservedRunningTime="2026-05-26 12:25:04.844929662 +0000 UTC m=+3467.207879330"
May 26 12:25:05.826566 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:05.826534    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:25:05.827534 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:05.827510    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 26 12:25:06.829576 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:06.829521    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 26 12:25:11.834357 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:11.834329    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:25:11.835135 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:11.835116    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:25:17.773672 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.773640    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"]
May 26 12:25:17.786338 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.773912    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kserve-container" containerID="cri-o://c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b" gracePeriod=30
May 26 12:25:17.786338 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.773959    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kube-rbac-proxy" containerID="cri-o://e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6" gracePeriod=30
May 26 12:25:17.893403 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.893373    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"]
May 26 12:25:17.893623 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.893612    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="storage-initializer"
May 26 12:25:17.898580 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.893625    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="storage-initializer"
May 26 12:25:17.898580 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.894066    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy"
May 26 12:25:17.898580 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.894083    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy"
May 26 12:25:17.898580 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.894236    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kserve-container"
May 26 12:25:17.898580 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.894271    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kserve-container"
May 26 12:25:17.898580 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.894546    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kube-rbac-proxy"
May 26 12:25:17.898580 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.894568    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9c3fe54-1747-41c3-ae38-be4bc280b15c" containerName="kserve-container"
May 26 12:25:17.914403 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.914375    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"]
May 26 12:25:17.914531 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.914496    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:17.917360 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.917339    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
May 26 12:25:17.917465 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:17.917449    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
May 26 12:25:18.066032 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.066001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.066032 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.066032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.066249 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.066055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.066249 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.066158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7c9\" (UniqueName: \"kubernetes.io/projected/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kube-api-access-qs7c9\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.167216 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.167184    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.167216 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.167216    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.167415 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.167240    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.167415 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.167273    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7c9\" (UniqueName: \"kubernetes.io/projected/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kube-api-access-qs7c9\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.167524 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.167505    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.169568 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.169552    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
May 26 12:25:18.169715 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.169699    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
May 26 12:25:18.176842 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.176820    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7c9\" (UniqueName: \"kubernetes.io/projected/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kube-api-access-qs7c9\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.177951 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.177922    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.179619 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.179604    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-cfnt8\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.224980 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.224952    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:18.416213 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.416174    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"]
May 26 12:25:18.419107 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:25:18.419077    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e89cf9_b6a6_41b9_8b37_69ccbaf117b9.slice/crio-1c7438680b7b6e2c161ad140e1fc316385d8955631b06cf86804535df56bcd40 WatchSource:0}: Error finding container 1c7438680b7b6e2c161ad140e1fc316385d8955631b06cf86804535df56bcd40: Status 404 returned error can't find the container with id 1c7438680b7b6e2c161ad140e1fc316385d8955631b06cf86804535df56bcd40
May 26 12:25:18.861884 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.861848    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" event={"ID":"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9","Type":"ContainerStarted","Data":"2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684"}
May 26 12:25:18.861884 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.861891    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" event={"ID":"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9","Type":"ContainerStarted","Data":"1c7438680b7b6e2c161ad140e1fc316385d8955631b06cf86804535df56bcd40"}
May 26 12:25:18.863623 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.863598    2563 generic.go:358] "Generic (PLEG): container finished" podID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerID="e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6" exitCode=2
May 26 12:25:18.863734 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:18.863670    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" event={"ID":"24b19391-e8b8-4d04-ae77-63fc264740dc","Type":"ContainerDied","Data":"e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6"}
May 26 12:25:20.333868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.333846    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:25:20.485589 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.485498    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b19391-e8b8-4d04-ae77-63fc264740dc-isvc-triton-kube-rbac-proxy-sar-config\") pod \"24b19391-e8b8-4d04-ae77-63fc264740dc\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") "
May 26 12:25:20.485589 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.485545    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b19391-e8b8-4d04-ae77-63fc264740dc-proxy-tls\") pod \"24b19391-e8b8-4d04-ae77-63fc264740dc\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") "
May 26 12:25:20.485589 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.485573    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgwls\" (UniqueName: \"kubernetes.io/projected/24b19391-e8b8-4d04-ae77-63fc264740dc-kube-api-access-vgwls\") pod \"24b19391-e8b8-4d04-ae77-63fc264740dc\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") "
May 26 12:25:20.485918 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.485682    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b19391-e8b8-4d04-ae77-63fc264740dc-kserve-provision-location\") pod \"24b19391-e8b8-4d04-ae77-63fc264740dc\" (UID: \"24b19391-e8b8-4d04-ae77-63fc264740dc\") "
May 26 12:25:20.485961 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.485936    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b19391-e8b8-4d04-ae77-63fc264740dc-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "24b19391-e8b8-4d04-ae77-63fc264740dc" (UID: "24b19391-e8b8-4d04-ae77-63fc264740dc"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:25:20.486060 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.486039    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b19391-e8b8-4d04-ae77-63fc264740dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24b19391-e8b8-4d04-ae77-63fc264740dc" (UID: "24b19391-e8b8-4d04-ae77-63fc264740dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:25:20.487777 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.487757    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b19391-e8b8-4d04-ae77-63fc264740dc-kube-api-access-vgwls" (OuterVolumeSpecName: "kube-api-access-vgwls") pod "24b19391-e8b8-4d04-ae77-63fc264740dc" (UID: "24b19391-e8b8-4d04-ae77-63fc264740dc"). InnerVolumeSpecName "kube-api-access-vgwls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:25:20.487854 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.487775    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b19391-e8b8-4d04-ae77-63fc264740dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24b19391-e8b8-4d04-ae77-63fc264740dc" (UID: "24b19391-e8b8-4d04-ae77-63fc264740dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:25:20.586261 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.586223    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24b19391-e8b8-4d04-ae77-63fc264740dc-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:25:20.586261 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.586254    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24b19391-e8b8-4d04-ae77-63fc264740dc-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:25:20.586261 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.586267    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24b19391-e8b8-4d04-ae77-63fc264740dc-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:25:20.586498 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.586279    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgwls\" (UniqueName: \"kubernetes.io/projected/24b19391-e8b8-4d04-ae77-63fc264740dc-kube-api-access-vgwls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:25:20.870420 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.870384    2563 generic.go:358] "Generic (PLEG): container finished" podID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerID="c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b" exitCode=0
May 26 12:25:20.870693 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.870462    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" event={"ID":"24b19391-e8b8-4d04-ae77-63fc264740dc","Type":"ContainerDied","Data":"c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b"}
May 26 12:25:20.870693 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.870476    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"
May 26 12:25:20.870693 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.870500    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl" event={"ID":"24b19391-e8b8-4d04-ae77-63fc264740dc","Type":"ContainerDied","Data":"99288c06b2cffe3fdb85948f6bb9d8b55c2b86454a68a73b4f408064009c9b6f"}
May 26 12:25:20.870693 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.870515    2563 scope.go:117] "RemoveContainer" containerID="e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6"
May 26 12:25:20.878649 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.878489    2563 scope.go:117] "RemoveContainer" containerID="c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b"
May 26 12:25:20.885484 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.885467    2563 scope.go:117] "RemoveContainer" containerID="451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215"
May 26 12:25:20.891485 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.891464    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"]
May 26 12:25:20.892782 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.892767    2563 scope.go:117] "RemoveContainer" containerID="e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6"
May 26 12:25:20.893050 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:25:20.893021    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6\": container with ID starting with e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6 not found: ID does not exist" containerID="e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6"
May 26 12:25:20.893106 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.893051    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6"} err="failed to get container status \"e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6\": rpc error: code = NotFound desc = could not find container \"e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6\": container with ID starting with e283d1381a3ab3545d9952ee903df78a55480f96d121c2916481d1de4c20cdc6 not found: ID does not exist"
May 26 12:25:20.893106 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.893071    2563 scope.go:117] "RemoveContainer" containerID="c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b"
May 26 12:25:20.893318 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:25:20.893303    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b\": container with ID starting with c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b not found: ID does not exist" containerID="c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b"
May 26 12:25:20.893372 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.893323    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b"} err="failed to get container status \"c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b\": rpc error: code = NotFound desc = could not find container \"c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b\": container with ID starting with c7a99f267b17b24d83dff924b0d4b7edadb5c8a58ee4a52f91548b1ffab02c0b not found: ID does not exist"
May 26 12:25:20.893372 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.893337    2563 scope.go:117] "RemoveContainer" containerID="451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215"
May 26 12:25:20.893564 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:25:20.893547    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215\": container with ID starting with 451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215 not found: ID does not exist" containerID="451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215"
May 26 12:25:20.893604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.893570    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215"} err="failed to get container status \"451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215\": rpc error: code = NotFound desc = could not find container \"451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215\": container with ID starting with 451be1abe896361be632a82dcfe5962988436e32a8cac0ee47faaa3c5323c215 not found: ID does not exist"
May 26 12:25:20.895121 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:20.895101    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-tf6gl"]
May 26 12:25:22.241956 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:22.241875    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" path="/var/lib/kubelet/pods/24b19391-e8b8-4d04-ae77-63fc264740dc/volumes"
May 26 12:25:22.878081 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:22.878049    2563 generic.go:358] "Generic (PLEG): container finished" podID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerID="2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684" exitCode=0
May 26 12:25:22.878251 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:22.878099    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" event={"ID":"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9","Type":"ContainerDied","Data":"2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684"}
May 26 12:25:42.844082 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:42.844052    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:25:42.934860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:42.934823    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" event={"ID":"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9","Type":"ContainerStarted","Data":"0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329"}
May 26 12:25:43.939021 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:43.938978    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" event={"ID":"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9","Type":"ContainerStarted","Data":"5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701"}
May 26 12:25:43.939418 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:43.939142    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:43.961715 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:43.961670    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podStartSLOduration=7.124566207 podStartE2EDuration="26.961656535s" podCreationTimestamp="2026-05-26 12:25:17 +0000 UTC" firstStartedPulling="2026-05-26 12:25:22.879264436 +0000 UTC m=+3485.242214084" lastFinishedPulling="2026-05-26 12:25:42.716354765 +0000 UTC m=+3505.079304412" observedRunningTime="2026-05-26 12:25:43.959614898 +0000 UTC m=+3506.322564567" watchObservedRunningTime="2026-05-26 12:25:43.961656535 +0000 UTC m=+3506.324606203"
May 26 12:25:44.942220 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:44.942186    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:44.943302 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:44.943274    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 12:25:45.945168 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:45.945126    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 12:25:50.949461 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:50.949430    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:25:50.950051 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:25:50.950026    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 12:26:00.950934 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:00.950893    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 12:26:10.950982 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:10.950944    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 12:26:20.950440 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:20.950402    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 12:26:30.950483 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:30.950443    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 12:26:40.950942 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:40.950914    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:26:48.141161 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.141118    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"]
May 26 12:26:48.141557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.141474    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" containerID="cri-o://0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329" gracePeriod=30
May 26 12:26:48.141557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.141493    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kube-rbac-proxy" containerID="cri-o://5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701" gracePeriod=30
May 26 12:26:48.351242 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351209    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"]
May 26 12:26:48.351461 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351450    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="storage-initializer"
May 26 12:26:48.351507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351463    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="storage-initializer"
May 26 12:26:48.351507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351473    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kube-rbac-proxy"
May 26 12:26:48.351507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351479    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kube-rbac-proxy"
May 26 12:26:48.351507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351492    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kserve-container"
May 26 12:26:48.351507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351498    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kserve-container"
May 26 12:26:48.351663 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351540    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kserve-container"
May 26 12:26:48.351663 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.351549    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="24b19391-e8b8-4d04-ae77-63fc264740dc" containerName="kube-rbac-proxy"
May 26 12:26:48.353299 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.353281    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.358696 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.358673    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
May 26 12:26:48.358848 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.358675    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 26 12:26:48.374285 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.374265    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"]
May 26 12:26:48.527438 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.527345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6k8\" (UniqueName: \"kubernetes.io/projected/8a61701a-6a34-4847-b092-6b7955ec28a3-kube-api-access-tn6k8\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.527438 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.527411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.527647 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.527440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a61701a-6a34-4847-b092-6b7955ec28a3-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.527647 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.527481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a61701a-6a34-4847-b092-6b7955ec28a3-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.628387 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.628353    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a61701a-6a34-4847-b092-6b7955ec28a3-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.628582 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.628412    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6k8\" (UniqueName: \"kubernetes.io/projected/8a61701a-6a34-4847-b092-6b7955ec28a3-kube-api-access-tn6k8\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.628582 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.628475    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.628582 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.628504    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a61701a-6a34-4847-b092-6b7955ec28a3-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.628762 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:26:48.628628    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
May 26 12:26:48.628762 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:26:48.628708    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls podName:8a61701a-6a34-4847-b092-6b7955ec28a3 nodeName:}" failed. No retries permitted until 2026-05-26 12:26:49.128685656 +0000 UTC m=+3571.491635305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" (UID: "8a61701a-6a34-4847-b092-6b7955ec28a3") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
May 26 12:26:48.628920 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.628787    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a61701a-6a34-4847-b092-6b7955ec28a3-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.629195 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.629171    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a61701a-6a34-4847-b092-6b7955ec28a3-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:48.638520 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:48.638496    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6k8\" (UniqueName: \"kubernetes.io/projected/8a61701a-6a34-4847-b092-6b7955ec28a3-kube-api-access-tn6k8\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:49.114277 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:49.114242    2563 generic.go:358] "Generic (PLEG): container finished" podID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerID="5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701" exitCode=2
May 26 12:26:49.114462 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:49.114313    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" event={"ID":"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9","Type":"ContainerDied","Data":"5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701"}
May 26 12:26:49.133769 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:49.133748    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:49.136257 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:49.136241    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:49.263308 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:49.263274    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:49.395181 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:49.395099    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"]
May 26 12:26:49.398138 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:26:49.398094    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a61701a_6a34_4847_b092_6b7955ec28a3.slice/crio-c6bce91fe2a42362f361ed1193a312b7dfe9fb6981122af5122b904b134dab2f WatchSource:0}: Error finding container c6bce91fe2a42362f361ed1193a312b7dfe9fb6981122af5122b904b134dab2f: Status 404 returned error can't find the container with id c6bce91fe2a42362f361ed1193a312b7dfe9fb6981122af5122b904b134dab2f
May 26 12:26:50.118475 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:50.118439    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" event={"ID":"8a61701a-6a34-4847-b092-6b7955ec28a3","Type":"ContainerStarted","Data":"94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e"}
May 26 12:26:50.118475 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:50.118479    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" event={"ID":"8a61701a-6a34-4847-b092-6b7955ec28a3","Type":"ContainerStarted","Data":"c6bce91fe2a42362f361ed1193a312b7dfe9fb6981122af5122b904b134dab2f"}
May 26 12:26:50.946369 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:50.946328    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 26 12:26:50.950733 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:50.950709    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 12:26:51.683106 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.683083    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:26:51.856666 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.856634    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") "
May 26 12:26:51.856969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.856686    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kserve-provision-location\") pod \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") "
May 26 12:26:51.856969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.856708    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-proxy-tls\") pod \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") "
May 26 12:26:51.856969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.856729    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs7c9\" (UniqueName: \"kubernetes.io/projected/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kube-api-access-qs7c9\") pod \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\" (UID: \"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9\") "
May 26 12:26:51.857157 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.857053    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" (UID: "16e89cf9-b6a6-41b9-8b37-69ccbaf117b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:26:51.857157 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.857080    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" (UID: "16e89cf9-b6a6-41b9-8b37-69ccbaf117b9"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:26:51.859010 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.858980    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kube-api-access-qs7c9" (OuterVolumeSpecName: "kube-api-access-qs7c9") pod "16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" (UID: "16e89cf9-b6a6-41b9-8b37-69ccbaf117b9"). InnerVolumeSpecName "kube-api-access-qs7c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:26:51.859099 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.858979    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" (UID: "16e89cf9-b6a6-41b9-8b37-69ccbaf117b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:26:51.957444 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.957412    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:26:51.957444 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.957440    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:26:51.957817 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.957453    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:26:51.957817 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:51.957462    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs7c9\" (UniqueName: \"kubernetes.io/projected/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9-kube-api-access-qs7c9\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:26:52.127388 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.127297    2563 generic.go:358] "Generic (PLEG): container finished" podID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerID="0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329" exitCode=0
May 26 12:26:52.127557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.127388    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"
May 26 12:26:52.127557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.127380    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" event={"ID":"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9","Type":"ContainerDied","Data":"0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329"}
May 26 12:26:52.127557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.127498    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8" event={"ID":"16e89cf9-b6a6-41b9-8b37-69ccbaf117b9","Type":"ContainerDied","Data":"1c7438680b7b6e2c161ad140e1fc316385d8955631b06cf86804535df56bcd40"}
May 26 12:26:52.127557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.127514    2563 scope.go:117] "RemoveContainer" containerID="5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701"
May 26 12:26:52.135876 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.135860    2563 scope.go:117] "RemoveContainer" containerID="0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329"
May 26 12:26:52.142359 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.142339    2563 scope.go:117] "RemoveContainer" containerID="2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684"
May 26 12:26:52.148613 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.148597    2563 scope.go:117] "RemoveContainer" containerID="5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701"
May 26 12:26:52.148861 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:26:52.148842    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701\": container with ID starting with 5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701 not found: ID does not exist" containerID="5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701"
May 26 12:26:52.148931 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.148871    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701"} err="failed to get container status \"5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701\": rpc error: code = NotFound desc = could not find container \"5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701\": container with ID starting with 5b5737f0698901e33aa717c02e475aceb6de822dad11fff3a1201682a62a3701 not found: ID does not exist"
May 26 12:26:52.148931 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.148889    2563 scope.go:117] "RemoveContainer" containerID="0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329"
May 26 12:26:52.149117 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:26:52.149102    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329\": container with ID starting with 0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329 not found: ID does not exist" containerID="0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329"
May 26 12:26:52.149163 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.149122    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329"} err="failed to get container status \"0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329\": rpc error: code = NotFound desc = could not find container \"0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329\": container with ID starting with 0dea927c2a6336d5582beb6c941b69ea063abb075da196706761c5b6e5138329 not found: ID does not exist"
May 26 12:26:52.149163 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.149136    2563 scope.go:117] "RemoveContainer" containerID="2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684"
May 26 12:26:52.149356 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:26:52.149333    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684\": container with ID starting with 2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684 not found: ID does not exist" containerID="2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684"
May 26 12:26:52.149401 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.149362    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684"} err="failed to get container status \"2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684\": rpc error: code = NotFound desc = could not find container \"2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684\": container with ID starting with 2e95e8c2809dd7c63e395278bc10aed012da6a86ffa556d6c968f6387fb5c684 not found: ID does not exist"
May 26 12:26:52.173476 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.173455    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"]
May 26 12:26:52.179200 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.179127    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-cfnt8"]
May 26 12:26:52.242293 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:52.242261    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" path="/var/lib/kubelet/pods/16e89cf9-b6a6-41b9-8b37-69ccbaf117b9/volumes"
May 26 12:26:53.132276 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:53.132248    2563 generic.go:358] "Generic (PLEG): container finished" podID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerID="94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e" exitCode=0
May 26 12:26:53.132648 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:53.132328    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" event={"ID":"8a61701a-6a34-4847-b092-6b7955ec28a3","Type":"ContainerDied","Data":"94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e"}
May 26 12:26:54.136424 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:54.136386    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" event={"ID":"8a61701a-6a34-4847-b092-6b7955ec28a3","Type":"ContainerStarted","Data":"8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763"}
May 26 12:26:54.136424 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:54.136419    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" event={"ID":"8a61701a-6a34-4847-b092-6b7955ec28a3","Type":"ContainerStarted","Data":"950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201"}
May 26 12:26:54.136905 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:54.136712    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:54.136905 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:54.136745    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:26:54.157605 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:26:54.157556    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" podStartSLOduration=6.157540916 podStartE2EDuration="6.157540916s" podCreationTimestamp="2026-05-26 12:26:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:26:54.157234706 +0000 UTC m=+3576.520184375" watchObservedRunningTime="2026-05-26 12:26:54.157540916 +0000 UTC m=+3576.520490587"
May 26 12:27:00.144892 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:00.144859    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:27:30.148949 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:30.148912    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:27:38.234754 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.234715    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"]
May 26 12:27:38.235255 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.235117    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kserve-container" containerID="cri-o://950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201" gracePeriod=30
May 26 12:27:38.235255 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.235179    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kube-rbac-proxy" containerID="cri-o://8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763" gracePeriod=30
May 26 12:27:38.354472 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354439    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"]
May 26 12:27:38.354810 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354777    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kube-rbac-proxy"
May 26 12:27:38.354859 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354811    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kube-rbac-proxy"
May 26 12:27:38.354859 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354828    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="storage-initializer"
May 26 12:27:38.354859 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354834    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="storage-initializer"
May 26 12:27:38.354859 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354843    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container"
May 26 12:27:38.354859 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354849    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container"
May 26 12:27:38.355017 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354901    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kube-rbac-proxy"
May 26 12:27:38.355017 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.354909    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="16e89cf9-b6a6-41b9-8b37-69ccbaf117b9" containerName="kserve-container"
May 26 12:27:38.356645 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.356623    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.358612 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.358589    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
May 26 12:27:38.358717 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.358592    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 26 12:27:38.368741 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.368720    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"]
May 26 12:27:38.383856 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.383828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c7d7b82-68e1-47c5-8756-f9126972b222-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.383995 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.383869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q289n\" (UniqueName: \"kubernetes.io/projected/8c7d7b82-68e1-47c5-8756-f9126972b222-kube-api-access-q289n\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.383995 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.383887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.384128 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.383996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c7d7b82-68e1-47c5-8756-f9126972b222-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.484842 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.484736    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q289n\" (UniqueName: \"kubernetes.io/projected/8c7d7b82-68e1-47c5-8756-f9126972b222-kube-api-access-q289n\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.484842 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.484774    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.484842 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.484836    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c7d7b82-68e1-47c5-8756-f9126972b222-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.485145 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.484900    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c7d7b82-68e1-47c5-8756-f9126972b222-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.485145 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:27:38.484932    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 26 12:27:38.485145 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:27:38.485004    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls podName:8c7d7b82-68e1-47c5-8756-f9126972b222 nodeName:}" failed. No retries permitted until 2026-05-26 12:27:38.984985433 +0000 UTC m=+3621.347935079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls") pod "xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" (UID: "8c7d7b82-68e1-47c5-8756-f9126972b222") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 26 12:27:38.485275 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.485248    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c7d7b82-68e1-47c5-8756-f9126972b222-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.485482 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.485463    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c7d7b82-68e1-47c5-8756-f9126972b222-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.493744 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.493724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q289n\" (UniqueName: \"kubernetes.io/projected/8c7d7b82-68e1-47c5-8756-f9126972b222-kube-api-access-q289n\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.988836 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.988782    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:38.991322 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:38.991300    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-5wbcp\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:39.259193 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:39.259099    2563 generic.go:358] "Generic (PLEG): container finished" podID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerID="8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763" exitCode=2
May 26 12:27:39.259193 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:39.259153    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" event={"ID":"8a61701a-6a34-4847-b092-6b7955ec28a3","Type":"ContainerDied","Data":"8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763"}
May 26 12:27:39.266430 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:39.266412    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:39.488310 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:39.488282    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"]
May 26 12:27:39.492313 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:27:39.492285    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7d7b82_68e1_47c5_8756_f9126972b222.slice/crio-2a35ca21f4d26d65fe75bdd399b6a3ec7bc21e13876bca673b466fe30637e445 WatchSource:0}: Error finding container 2a35ca21f4d26d65fe75bdd399b6a3ec7bc21e13876bca673b466fe30637e445: Status 404 returned error can't find the container with id 2a35ca21f4d26d65fe75bdd399b6a3ec7bc21e13876bca673b466fe30637e445
May 26 12:27:40.139524 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:40.139478    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 26 12:27:40.262471 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:40.262433    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" event={"ID":"8c7d7b82-68e1-47c5-8756-f9126972b222","Type":"ContainerStarted","Data":"c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6"}
May 26 12:27:40.262471 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:40.262469    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" event={"ID":"8c7d7b82-68e1-47c5-8756-f9126972b222","Type":"ContainerStarted","Data":"2a35ca21f4d26d65fe75bdd399b6a3ec7bc21e13876bca673b466fe30637e445"}
May 26 12:27:43.271983 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:43.271953    2563 generic.go:358] "Generic (PLEG): container finished" podID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerID="c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6" exitCode=0
May 26 12:27:43.272287 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:43.271994    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" event={"ID":"8c7d7b82-68e1-47c5-8756-f9126972b222","Type":"ContainerDied","Data":"c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6"}
May 26 12:27:44.276349 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.276312    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" event={"ID":"8c7d7b82-68e1-47c5-8756-f9126972b222","Type":"ContainerStarted","Data":"5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755"}
May 26 12:27:44.276349 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.276344    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" event={"ID":"8c7d7b82-68e1-47c5-8756-f9126972b222","Type":"ContainerStarted","Data":"51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f"}
May 26 12:27:44.276753 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.276553    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:44.297153 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.297111    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" podStartSLOduration=6.297098859 podStartE2EDuration="6.297098859s" podCreationTimestamp="2026-05-26 12:27:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:27:44.295981737 +0000 UTC m=+3626.658931406" watchObservedRunningTime="2026-05-26 12:27:44.297098859 +0000 UTC m=+3626.660048527"
May 26 12:27:44.773309 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.773265    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:27:44.829843 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.829808    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a61701a-6a34-4847-b092-6b7955ec28a3-kserve-provision-location\") pod \"8a61701a-6a34-4847-b092-6b7955ec28a3\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") "
May 26 12:27:44.830000 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.829864    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls\") pod \"8a61701a-6a34-4847-b092-6b7955ec28a3\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") "
May 26 12:27:44.830000 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.829893    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn6k8\" (UniqueName: \"kubernetes.io/projected/8a61701a-6a34-4847-b092-6b7955ec28a3-kube-api-access-tn6k8\") pod \"8a61701a-6a34-4847-b092-6b7955ec28a3\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") "
May 26 12:27:44.830000 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.829919    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a61701a-6a34-4847-b092-6b7955ec28a3-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"8a61701a-6a34-4847-b092-6b7955ec28a3\" (UID: \"8a61701a-6a34-4847-b092-6b7955ec28a3\") "
May 26 12:27:44.830247 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.830220    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a61701a-6a34-4847-b092-6b7955ec28a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a61701a-6a34-4847-b092-6b7955ec28a3" (UID: "8a61701a-6a34-4847-b092-6b7955ec28a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:27:44.830336 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.830310    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a61701a-6a34-4847-b092-6b7955ec28a3-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "8a61701a-6a34-4847-b092-6b7955ec28a3" (UID: "8a61701a-6a34-4847-b092-6b7955ec28a3"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:27:44.832138 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.832119    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a61701a-6a34-4847-b092-6b7955ec28a3" (UID: "8a61701a-6a34-4847-b092-6b7955ec28a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:27:44.832199 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.832137    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a61701a-6a34-4847-b092-6b7955ec28a3-kube-api-access-tn6k8" (OuterVolumeSpecName: "kube-api-access-tn6k8") pod "8a61701a-6a34-4847-b092-6b7955ec28a3" (UID: "8a61701a-6a34-4847-b092-6b7955ec28a3"). InnerVolumeSpecName "kube-api-access-tn6k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:27:44.930634 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.930563    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a61701a-6a34-4847-b092-6b7955ec28a3-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:27:44.930634 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.930591    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a61701a-6a34-4847-b092-6b7955ec28a3-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:27:44.930634 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.930601    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6k8\" (UniqueName: \"kubernetes.io/projected/8a61701a-6a34-4847-b092-6b7955ec28a3-kube-api-access-tn6k8\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:27:44.930634 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:44.930610    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a61701a-6a34-4847-b092-6b7955ec28a3-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:27:45.280834 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.280724    2563 generic.go:358] "Generic (PLEG): container finished" podID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerID="950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201" exitCode=0
May 26 12:27:45.280834 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.280825    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" event={"ID":"8a61701a-6a34-4847-b092-6b7955ec28a3","Type":"ContainerDied","Data":"950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201"}
May 26 12:27:45.281349 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.280848    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"
May 26 12:27:45.281349 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.280864    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8" event={"ID":"8a61701a-6a34-4847-b092-6b7955ec28a3","Type":"ContainerDied","Data":"c6bce91fe2a42362f361ed1193a312b7dfe9fb6981122af5122b904b134dab2f"}
May 26 12:27:45.281349 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.280884    2563 scope.go:117] "RemoveContainer" containerID="8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763"
May 26 12:27:45.281505 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.281375    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:27:45.288177 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.288161    2563 scope.go:117] "RemoveContainer" containerID="950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201"
May 26 12:27:45.295414 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.295396    2563 scope.go:117] "RemoveContainer" containerID="94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e"
May 26 12:27:45.301196 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.301175    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"]
May 26 12:27:45.304099 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.304079    2563 scope.go:117] "RemoveContainer" containerID="8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763"
May 26 12:27:45.304392 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:27:45.304367    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763\": container with ID starting with 8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763 not found: ID does not exist" containerID="8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763"
May 26 12:27:45.304497 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.304404    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763"} err="failed to get container status \"8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763\": rpc error: code = NotFound desc = could not find container \"8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763\": container with ID starting with 8a8ca57f0d1c68358f1ea141c4072547acda06d935abadaa73ca3b6554179763 not found: ID does not exist"
May 26 12:27:45.304497 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.304428    2563 scope.go:117] "RemoveContainer" containerID="950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201"
May 26 12:27:45.304625 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.304582    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hbpl8"]
May 26 12:27:45.304736 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:27:45.304718    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201\": container with ID starting with 950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201 not found: ID does not exist" containerID="950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201"
May 26 12:27:45.305114 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.304740    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201"} err="failed to get container status \"950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201\": rpc error: code = NotFound desc = could not find container \"950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201\": container with ID starting with 950159b96665e60b972267327eadc0d45c949aa0812afb6a352855a0027bb201 not found: ID does not exist"
May 26 12:27:45.305114 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.304756    2563 scope.go:117] "RemoveContainer" containerID="94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e"
May 26 12:27:45.305114 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:27:45.305035    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e\": container with ID starting with 94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e not found: ID does not exist" containerID="94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e"
May 26 12:27:45.305114 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:45.305064    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e"} err="failed to get container status \"94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e\": rpc error: code = NotFound desc = could not find container \"94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e\": container with ID starting with 94bf4de1644e29bdc647364fb9fb95fce9936eef1b17100c5756f291a315434e not found: ID does not exist"
May 26 12:27:46.242323 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:46.242288    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" path="/var/lib/kubelet/pods/8a61701a-6a34-4847-b092-6b7955ec28a3/volumes"
May 26 12:27:51.289500 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:27:51.289419    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:28:01.478315 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:01.478285    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:28:01.485635 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:01.485609    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:28:21.293050 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:21.293015    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:28:28.399443 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.399411    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"]
May 26 12:28:28.399867 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.399703    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kserve-container" containerID="cri-o://51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f" gracePeriod=30
May 26 12:28:28.399867 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.399747    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kube-rbac-proxy" containerID="cri-o://5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755" gracePeriod=30
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.489485    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"]
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.490078    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="storage-initializer"
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.490098    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="storage-initializer"
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.490130    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kserve-container"
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.490139    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kserve-container"
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.490173    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kube-rbac-proxy"
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.490182    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kube-rbac-proxy"
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.490291    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kube-rbac-proxy"
May 26 12:28:28.491967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.490301    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8a61701a-6a34-4847-b092-6b7955ec28a3" containerName="kserve-container"
May 26 12:28:28.493896 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.493870    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.495943 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.495913    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
May 26 12:28:28.496059 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.495959    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
May 26 12:28:28.500996 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.500975    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"]
May 26 12:28:28.654927 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.654838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr85r\" (UniqueName: \"kubernetes.io/projected/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kube-api-access-jr85r\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.654927 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.654890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.654927 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.654910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.655152 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.654982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.756160 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.756129    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jr85r\" (UniqueName: \"kubernetes.io/projected/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kube-api-access-jr85r\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.756311 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.756177    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.756311 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.756198    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.756311 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.756225    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.756470 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:28:28.756323    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
May 26 12:28:28.756470 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:28:28.756388    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls podName:f55df3f4-a0e1-456b-b3b5-0de03fb8223b nodeName:}" failed. No retries permitted until 2026-05-26 12:28:29.256373304 +0000 UTC m=+3671.619322951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls") pod "isvc-xgboost-runtime-predictor-779db84d9-jwvsm" (UID: "f55df3f4-a0e1-456b-b3b5-0de03fb8223b") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
May 26 12:28:28.756683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.756660    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.756848 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.756826    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:28.764508 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:28.764486    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr85r\" (UniqueName: \"kubernetes.io/projected/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kube-api-access-jr85r\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:29.259318 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:29.259277    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:29.261813 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:29.261775    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-jwvsm\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:29.397063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:29.397029    2563 generic.go:358] "Generic (PLEG): container finished" podID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerID="5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755" exitCode=2
May 26 12:28:29.397063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:29.397067    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" event={"ID":"8c7d7b82-68e1-47c5-8756-f9126972b222","Type":"ContainerDied","Data":"5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755"}
May 26 12:28:29.404162 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:29.404145    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:29.529938 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:29.529852    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"]
May 26 12:28:29.533066 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:28:29.533040    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55df3f4_a0e1_456b_b3b5_0de03fb8223b.slice/crio-593527c6ab995b072df9d7c7c6045f50c79a241f27bdad043009276da3e8aa3a WatchSource:0}: Error finding container 593527c6ab995b072df9d7c7c6045f50c79a241f27bdad043009276da3e8aa3a: Status 404 returned error can't find the container with id 593527c6ab995b072df9d7c7c6045f50c79a241f27bdad043009276da3e8aa3a
May 26 12:28:30.401482 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:30.401443    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" event={"ID":"f55df3f4-a0e1-456b-b3b5-0de03fb8223b","Type":"ContainerStarted","Data":"2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd"}
May 26 12:28:30.401482 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:30.401487    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" event={"ID":"f55df3f4-a0e1-456b-b3b5-0de03fb8223b","Type":"ContainerStarted","Data":"593527c6ab995b072df9d7c7c6045f50c79a241f27bdad043009276da3e8aa3a"}
May 26 12:28:31.285219 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:31.285174    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 26 12:28:31.290640 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:31.290610    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
May 26 12:28:34.413195 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.413156    2563 generic.go:358] "Generic (PLEG): container finished" podID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerID="2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd" exitCode=0
May 26 12:28:34.413195 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.413201    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" event={"ID":"f55df3f4-a0e1-456b-b3b5-0de03fb8223b","Type":"ContainerDied","Data":"2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd"}
May 26 12:28:34.746166 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.746137    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:28:34.797998 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.797973    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c7d7b82-68e1-47c5-8756-f9126972b222-kserve-provision-location\") pod \"8c7d7b82-68e1-47c5-8756-f9126972b222\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") "
May 26 12:28:34.798147 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.798033    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c7d7b82-68e1-47c5-8756-f9126972b222-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"8c7d7b82-68e1-47c5-8756-f9126972b222\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") "
May 26 12:28:34.798147 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.798065    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls\") pod \"8c7d7b82-68e1-47c5-8756-f9126972b222\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") "
May 26 12:28:34.798147 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.798081    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q289n\" (UniqueName: \"kubernetes.io/projected/8c7d7b82-68e1-47c5-8756-f9126972b222-kube-api-access-q289n\") pod \"8c7d7b82-68e1-47c5-8756-f9126972b222\" (UID: \"8c7d7b82-68e1-47c5-8756-f9126972b222\") "
May 26 12:28:34.798334 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.798264    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7d7b82-68e1-47c5-8756-f9126972b222-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c7d7b82-68e1-47c5-8756-f9126972b222" (UID: "8c7d7b82-68e1-47c5-8756-f9126972b222"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:28:34.798452 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.798430    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7d7b82-68e1-47c5-8756-f9126972b222-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "8c7d7b82-68e1-47c5-8756-f9126972b222" (UID: "8c7d7b82-68e1-47c5-8756-f9126972b222"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:28:34.800288 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.800260    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c7d7b82-68e1-47c5-8756-f9126972b222" (UID: "8c7d7b82-68e1-47c5-8756-f9126972b222"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:28:34.800382 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.800316    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7d7b82-68e1-47c5-8756-f9126972b222-kube-api-access-q289n" (OuterVolumeSpecName: "kube-api-access-q289n") pod "8c7d7b82-68e1-47c5-8756-f9126972b222" (UID: "8c7d7b82-68e1-47c5-8756-f9126972b222"). InnerVolumeSpecName "kube-api-access-q289n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:28:34.898822 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.898750    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c7d7b82-68e1-47c5-8756-f9126972b222-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:28:34.898822 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.898816    2563 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c7d7b82-68e1-47c5-8756-f9126972b222-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:28:34.898822 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.898832    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c7d7b82-68e1-47c5-8756-f9126972b222-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:28:34.899048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:34.898847    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q289n\" (UniqueName: \"kubernetes.io/projected/8c7d7b82-68e1-47c5-8756-f9126972b222-kube-api-access-q289n\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:28:35.417582 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.417546    2563 generic.go:358] "Generic (PLEG): container finished" podID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerID="51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f" exitCode=0
May 26 12:28:35.418063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.417637    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"
May 26 12:28:35.418063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.417636    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" event={"ID":"8c7d7b82-68e1-47c5-8756-f9126972b222","Type":"ContainerDied","Data":"51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f"}
May 26 12:28:35.418063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.417681    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp" event={"ID":"8c7d7b82-68e1-47c5-8756-f9126972b222","Type":"ContainerDied","Data":"2a35ca21f4d26d65fe75bdd399b6a3ec7bc21e13876bca673b466fe30637e445"}
May 26 12:28:35.418063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.417701    2563 scope.go:117] "RemoveContainer" containerID="5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755"
May 26 12:28:35.419686 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.419663    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" event={"ID":"f55df3f4-a0e1-456b-b3b5-0de03fb8223b","Type":"ContainerStarted","Data":"457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3"}
May 26 12:28:35.419819 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.419701    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" event={"ID":"f55df3f4-a0e1-456b-b3b5-0de03fb8223b","Type":"ContainerStarted","Data":"340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46"}
May 26 12:28:35.419935 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.419907    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:35.430372 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.430352    2563 scope.go:117] "RemoveContainer" containerID="51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f"
May 26 12:28:35.437162 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.437145    2563 scope.go:117] "RemoveContainer" containerID="c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6"
May 26 12:28:35.442531 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.442493    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podStartSLOduration=7.442482794 podStartE2EDuration="7.442482794s" podCreationTimestamp="2026-05-26 12:28:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:28:35.441232175 +0000 UTC m=+3677.804181846" watchObservedRunningTime="2026-05-26 12:28:35.442482794 +0000 UTC m=+3677.805432462"
May 26 12:28:35.443941 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.443923    2563 scope.go:117] "RemoveContainer" containerID="5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755"
May 26 12:28:35.444207 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:28:35.444188    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755\": container with ID starting with 5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755 not found: ID does not exist" containerID="5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755"
May 26 12:28:35.444258 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.444215    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755"} err="failed to get container status \"5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755\": rpc error: code = NotFound desc = could not find container \"5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755\": container with ID starting with 5036706c640e77d2769e61494caa704375849100707098a1562326a8c5a0d755 not found: ID does not exist"
May 26 12:28:35.444258 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.444230    2563 scope.go:117] "RemoveContainer" containerID="51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f"
May 26 12:28:35.444498 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:28:35.444472    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f\": container with ID starting with 51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f not found: ID does not exist" containerID="51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f"
May 26 12:28:35.444568 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.444510    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f"} err="failed to get container status \"51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f\": rpc error: code = NotFound desc = could not find container \"51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f\": container with ID starting with 51724483aea406f3c7dbb943e830f2c6b1e989b38cd413f46e685efea7a4a99f not found: ID does not exist"
May 26 12:28:35.444568 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.444531    2563 scope.go:117] "RemoveContainer" containerID="c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6"
May 26 12:28:35.444786 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:28:35.444753    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6\": container with ID starting with c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6 not found: ID does not exist" containerID="c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6"
May 26 12:28:35.444880 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.444815    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6"} err="failed to get container status \"c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6\": rpc error: code = NotFound desc = could not find container \"c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6\": container with ID starting with c3e0f9239e7a954d42aad84c0d4c9194f6950e8ab0ad1fe7f475d076774dcbe6 not found: ID does not exist"
May 26 12:28:35.454868 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.454847    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"]
May 26 12:28:35.456181 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:35.456161    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5wbcp"]
May 26 12:28:36.242018 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:36.241973    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" path="/var/lib/kubelet/pods/8c7d7b82-68e1-47c5-8756-f9126972b222/volumes"
May 26 12:28:36.423439 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:36.423410    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:36.424625 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:36.424599    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 12:28:37.426236 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:37.426199    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 12:28:42.430341 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:42.430306    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:28:42.430936 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:42.430909    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 12:28:52.431728 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:28:52.431686    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 12:29:02.431620 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:02.431573    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 12:29:12.431045 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:12.430997    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 12:29:22.431728 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:22.431684    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 12:29:32.431748 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:32.431719    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:29:38.560747 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.560716    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"]
May 26 12:29:38.561148 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.561047    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container" containerID="cri-o://340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46" gracePeriod=30
May 26 12:29:38.561148 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.561074    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kube-rbac-proxy" containerID="cri-o://457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3" gracePeriod=30
May 26 12:29:38.656101 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656071    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"]
May 26 12:29:38.656411 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656394    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kube-rbac-proxy"
May 26 12:29:38.656494 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656414    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kube-rbac-proxy"
May 26 12:29:38.656494 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656435    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kserve-container"
May 26 12:29:38.656494 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656443    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kserve-container"
May 26 12:29:38.656494 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656458    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="storage-initializer"
May 26 12:29:38.656494 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656468    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="storage-initializer"
May 26 12:29:38.656757 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656568    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kserve-container"
May 26 12:29:38.656757 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.656584    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c7d7b82-68e1-47c5-8756-f9126972b222" containerName="kube-rbac-proxy"
May 26 12:29:38.658490 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.658471    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.660345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.660316    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
May 26 12:29:38.660473 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.660455    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
May 26 12:29:38.670162 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.670142    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"]
May 26 12:29:38.707827 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.707787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.707944 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.707841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cg7\" (UniqueName: \"kubernetes.io/projected/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kube-api-access-t5cg7\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.707944 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.707864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.708021 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.707953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.808820 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.808770    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cg7\" (UniqueName: \"kubernetes.io/projected/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kube-api-access-t5cg7\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.808820 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.808825    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.809056 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.808862    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.809056 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.808894    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.809056 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:29:38.808997    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
May 26 12:29:38.809218 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:29:38.809080    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls podName:eeb8b40a-8a5b-42d4-9c0a-cc706de13de0 nodeName:}" failed. No retries permitted until 2026-05-26 12:29:39.309063283 +0000 UTC m=+3741.672012931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" (UID: "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
May 26 12:29:38.809292 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.809227    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.809567 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.809545    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:38.819618 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:38.819557    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cg7\" (UniqueName: \"kubernetes.io/projected/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kube-api-access-t5cg7\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:39.313426 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:39.313398    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:39.316022 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:39.316000    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:39.568394 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:39.568300    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:39.594856 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:39.594824    2563 generic.go:358] "Generic (PLEG): container finished" podID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerID="457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3" exitCode=2
May 26 12:29:39.595020 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:39.594894    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" event={"ID":"f55df3f4-a0e1-456b-b3b5-0de03fb8223b","Type":"ContainerDied","Data":"457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3"}
May 26 12:29:39.694699 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:39.694668    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"]
May 26 12:29:39.697572 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:29:39.697531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb8b40a_8a5b_42d4_9c0a_cc706de13de0.slice/crio-aac11c86e900cf6329e6dc5d33f27473255c40ae97bea8ff5c751cd28835559a WatchSource:0}: Error finding container aac11c86e900cf6329e6dc5d33f27473255c40ae97bea8ff5c751cd28835559a: Status 404 returned error can't find the container with id aac11c86e900cf6329e6dc5d33f27473255c40ae97bea8ff5c751cd28835559a
May 26 12:29:40.599314 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:40.599279    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" event={"ID":"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0","Type":"ContainerStarted","Data":"8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46"}
May 26 12:29:40.599314 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:40.599317    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" event={"ID":"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0","Type":"ContainerStarted","Data":"aac11c86e900cf6329e6dc5d33f27473255c40ae97bea8ff5c751cd28835559a"}
May 26 12:29:42.102686 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.102664    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:29:42.133370 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.133340    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr85r\" (UniqueName: \"kubernetes.io/projected/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kube-api-access-jr85r\") pod \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") "
May 26 12:29:42.133507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.133397    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kserve-provision-location\") pod \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") "
May 26 12:29:42.133507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.133439    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") "
May 26 12:29:42.133507 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.133482    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls\") pod \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\" (UID: \"f55df3f4-a0e1-456b-b3b5-0de03fb8223b\") "
May 26 12:29:42.133780 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.133746    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f55df3f4-a0e1-456b-b3b5-0de03fb8223b" (UID: "f55df3f4-a0e1-456b-b3b5-0de03fb8223b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:29:42.133920 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.133782    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "f55df3f4-a0e1-456b-b3b5-0de03fb8223b" (UID: "f55df3f4-a0e1-456b-b3b5-0de03fb8223b"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:29:42.135571 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.135543    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kube-api-access-jr85r" (OuterVolumeSpecName: "kube-api-access-jr85r") pod "f55df3f4-a0e1-456b-b3b5-0de03fb8223b" (UID: "f55df3f4-a0e1-456b-b3b5-0de03fb8223b"). InnerVolumeSpecName "kube-api-access-jr85r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:29:42.135650 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.135609    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f55df3f4-a0e1-456b-b3b5-0de03fb8223b" (UID: "f55df3f4-a0e1-456b-b3b5-0de03fb8223b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:29:42.234171 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.234089    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:29:42.234171 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.234123    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:29:42.234171 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.234133    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:29:42.234171 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.234144    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr85r\" (UniqueName: \"kubernetes.io/projected/f55df3f4-a0e1-456b-b3b5-0de03fb8223b-kube-api-access-jr85r\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:29:42.606388 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.606356    2563 generic.go:358] "Generic (PLEG): container finished" podID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerID="340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46" exitCode=0
May 26 12:29:42.606388 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.606396    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" event={"ID":"f55df3f4-a0e1-456b-b3b5-0de03fb8223b","Type":"ContainerDied","Data":"340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46"}
May 26 12:29:42.606654 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.606421    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm" event={"ID":"f55df3f4-a0e1-456b-b3b5-0de03fb8223b","Type":"ContainerDied","Data":"593527c6ab995b072df9d7c7c6045f50c79a241f27bdad043009276da3e8aa3a"}
May 26 12:29:42.606654 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.606437    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"
May 26 12:29:42.606654 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.606442    2563 scope.go:117] "RemoveContainer" containerID="457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3"
May 26 12:29:42.614170 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.614150    2563 scope.go:117] "RemoveContainer" containerID="340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46"
May 26 12:29:42.620711 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.620696    2563 scope.go:117] "RemoveContainer" containerID="2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd"
May 26 12:29:42.622771 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.622752    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"]
May 26 12:29:42.626731 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.626709    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-jwvsm"]
May 26 12:29:42.627845 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.627828    2563 scope.go:117] "RemoveContainer" containerID="457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3"
May 26 12:29:42.628091 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:29:42.628073    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3\": container with ID starting with 457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3 not found: ID does not exist" containerID="457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3"
May 26 12:29:42.628149 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.628098    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3"} err="failed to get container status \"457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3\": rpc error: code = NotFound desc = could not find container \"457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3\": container with ID starting with 457da11141e3582c5bd4ed252e01a06bd2b156c38d9bba27bd885104f621bab3 not found: ID does not exist"
May 26 12:29:42.628149 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.628115    2563 scope.go:117] "RemoveContainer" containerID="340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46"
May 26 12:29:42.628313 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:29:42.628299    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46\": container with ID starting with 340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46 not found: ID does not exist" containerID="340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46"
May 26 12:29:42.628359 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.628316    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46"} err="failed to get container status \"340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46\": rpc error: code = NotFound desc = could not find container \"340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46\": container with ID starting with 340f2f8c3f645949506127fcbb1e8258aff2080272241572e94efbe03ad70f46 not found: ID does not exist"
May 26 12:29:42.628359 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.628328    2563 scope.go:117] "RemoveContainer" containerID="2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd"
May 26 12:29:42.628519 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:29:42.628501    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd\": container with ID starting with 2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd not found: ID does not exist" containerID="2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd"
May 26 12:29:42.628558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:42.628524    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd"} err="failed to get container status \"2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd\": rpc error: code = NotFound desc = could not find container \"2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd\": container with ID starting with 2ac27e6decc0251a1755c04a4d852a1aeb51576ebbca272984eaf6391ae5c5cd not found: ID does not exist"
May 26 12:29:43.611292 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:43.611258    2563 generic.go:358] "Generic (PLEG): container finished" podID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerID="8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46" exitCode=0
May 26 12:29:43.611647 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:43.611328    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" event={"ID":"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0","Type":"ContainerDied","Data":"8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46"}
May 26 12:29:44.242257 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:44.242227    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" path="/var/lib/kubelet/pods/f55df3f4-a0e1-456b-b3b5-0de03fb8223b/volumes"
May 26 12:29:44.615859 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:44.615819    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" event={"ID":"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0","Type":"ContainerStarted","Data":"e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc"}
May 26 12:29:44.616276 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:44.615863    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" event={"ID":"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0","Type":"ContainerStarted","Data":"d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a"}
May 26 12:29:44.616276 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:44.616079    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:44.637946 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:44.637901    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" podStartSLOduration=6.637887893 podStartE2EDuration="6.637887893s" podCreationTimestamp="2026-05-26 12:29:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:29:44.636328363 +0000 UTC m=+3746.999278032" watchObservedRunningTime="2026-05-26 12:29:44.637887893 +0000 UTC m=+3747.000837561"
May 26 12:29:45.619397 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:45.619364    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:29:51.626834 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:29:51.626804    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:30:21.701302 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:21.701241    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 26 12:30:31.629609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:31.629578    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:30:38.767589 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.767557    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"]
May 26 12:30:38.768029 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.767897    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kserve-container" containerID="cri-o://d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a" gracePeriod=30
May 26 12:30:38.768029 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.767947    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kube-rbac-proxy" containerID="cri-o://e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc" gracePeriod=30
May 26 12:30:38.854567 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854532    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"]
May 26 12:30:38.854832 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854820    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container"
May 26 12:30:38.854882 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854834    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container"
May 26 12:30:38.854882 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854845    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kube-rbac-proxy"
May 26 12:30:38.854882 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854853    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kube-rbac-proxy"
May 26 12:30:38.854882 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854868    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="storage-initializer"
May 26 12:30:38.854882 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854877    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="storage-initializer"
May 26 12:30:38.855045 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854924    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kserve-container"
May 26 12:30:38.855045 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.854936    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="f55df3f4-a0e1-456b-b3b5-0de03fb8223b" containerName="kube-rbac-proxy"
May 26 12:30:38.862479 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.858193    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:38.862891 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.862873    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
May 26 12:30:38.863002 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.862934    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
May 26 12:30:38.869479 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.869453    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"]
May 26 12:30:38.935544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.935507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbh6\" (UniqueName: \"kubernetes.io/projected/7550d808-292d-4897-b71e-db04dada687b-kube-api-access-8wbh6\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:38.935701 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.935562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:38.935701 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.935626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7550d808-292d-4897-b71e-db04dada687b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:38.935701 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:38.935665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7550d808-292d-4897-b71e-db04dada687b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.036579 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.036499    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7550d808-292d-4897-b71e-db04dada687b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.036579 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.036541    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7550d808-292d-4897-b71e-db04dada687b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.036830 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.036596    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbh6\" (UniqueName: \"kubernetes.io/projected/7550d808-292d-4897-b71e-db04dada687b-kube-api-access-8wbh6\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.036830 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.036632    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.036830 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:30:39.036786    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
May 26 12:30:39.037001 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:30:39.036890    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls podName:7550d808-292d-4897-b71e-db04dada687b nodeName:}" failed. No retries permitted until 2026-05-26 12:30:39.536869392 +0000 UTC m=+3801.899819039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls") pod "isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" (UID: "7550d808-292d-4897-b71e-db04dada687b") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
May 26 12:30:39.037113 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.037088    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7550d808-292d-4897-b71e-db04dada687b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.037261 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.037243    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7550d808-292d-4897-b71e-db04dada687b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.047623 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.047591    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbh6\" (UniqueName: \"kubernetes.io/projected/7550d808-292d-4897-b71e-db04dada687b-kube-api-access-8wbh6\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.540979 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.540938    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.543435 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.543415    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.761684 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.761651    2563 generic.go:358] "Generic (PLEG): container finished" podID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerID="e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc" exitCode=2
May 26 12:30:39.761684 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.761689    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" event={"ID":"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0","Type":"ContainerDied","Data":"e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc"}
May 26 12:30:39.772983 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.772966    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:39.897896 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:39.897858    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"]
May 26 12:30:39.900889 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:30:39.900857    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7550d808_292d_4897_b71e_db04dada687b.slice/crio-603e0315aa644ce8c207e3c9eb5ba334ed0d228701e6d10cbf59995e53618d68 WatchSource:0}: Error finding container 603e0315aa644ce8c207e3c9eb5ba334ed0d228701e6d10cbf59995e53618d68: Status 404 returned error can't find the container with id 603e0315aa644ce8c207e3c9eb5ba334ed0d228701e6d10cbf59995e53618d68
May 26 12:30:40.765946 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:40.765905    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" event={"ID":"7550d808-292d-4897-b71e-db04dada687b","Type":"ContainerStarted","Data":"f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf"}
May 26 12:30:40.765946 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:40.765942    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" event={"ID":"7550d808-292d-4897-b71e-db04dada687b","Type":"ContainerStarted","Data":"603e0315aa644ce8c207e3c9eb5ba334ed0d228701e6d10cbf59995e53618d68"}
May 26 12:30:41.623040 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:41.622989    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
May 26 12:30:43.775237 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:43.775150    2563 generic.go:358] "Generic (PLEG): container finished" podID="7550d808-292d-4897-b71e-db04dada687b" containerID="f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf" exitCode=0
May 26 12:30:43.775237 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:43.775213    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" event={"ID":"7550d808-292d-4897-b71e-db04dada687b","Type":"ContainerDied","Data":"f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf"}
May 26 12:30:43.776391 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:43.776371    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:30:44.780020 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:44.779986    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" event={"ID":"7550d808-292d-4897-b71e-db04dada687b","Type":"ContainerStarted","Data":"a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844"}
May 26 12:30:44.780020 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:44.780025    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" event={"ID":"7550d808-292d-4897-b71e-db04dada687b","Type":"ContainerStarted","Data":"9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651"}
May 26 12:30:44.780443 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:44.780216    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:44.798314 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:44.798267    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podStartSLOduration=6.798250963 podStartE2EDuration="6.798250963s" podCreationTimestamp="2026-05-26 12:30:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:30:44.797880158 +0000 UTC m=+3807.160829828" watchObservedRunningTime="2026-05-26 12:30:44.798250963 +0000 UTC m=+3807.161200633"
May 26 12:30:45.782776 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:45.782747    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:45.783913 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:45.783884    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 12:30:46.110301 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.110278    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:30:46.293976 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.293952    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kserve-provision-location\") pod \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") "
May 26 12:30:46.294124 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.293990    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5cg7\" (UniqueName: \"kubernetes.io/projected/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kube-api-access-t5cg7\") pod \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") "
May 26 12:30:46.294124 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.294042    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") "
May 26 12:30:46.294124 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.294086    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls\") pod \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\" (UID: \"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0\") "
May 26 12:30:46.294312 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.294243    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" (UID: "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:30:46.294384 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.294371    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" (UID: "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:30:46.296175 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.296148    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" (UID: "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:30:46.296274 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.296238    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kube-api-access-t5cg7" (OuterVolumeSpecName: "kube-api-access-t5cg7") pod "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" (UID: "eeb8b40a-8a5b-42d4-9c0a-cc706de13de0"). InnerVolumeSpecName "kube-api-access-t5cg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:30:46.395143 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.395115    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:30:46.395143 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.395140    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:30:46.395143 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.395150    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:30:46.395357 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.395158    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5cg7\" (UniqueName: \"kubernetes.io/projected/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0-kube-api-access-t5cg7\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:30:46.787070 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.786963    2563 generic.go:358] "Generic (PLEG): container finished" podID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerID="d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a" exitCode=0
May 26 12:30:46.787070 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.787062    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"
May 26 12:30:46.787070 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.787047    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" event={"ID":"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0","Type":"ContainerDied","Data":"d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a"}
May 26 12:30:46.787582 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.787106    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm" event={"ID":"eeb8b40a-8a5b-42d4-9c0a-cc706de13de0","Type":"ContainerDied","Data":"aac11c86e900cf6329e6dc5d33f27473255c40ae97bea8ff5c751cd28835559a"}
May 26 12:30:46.787582 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.787127    2563 scope.go:117] "RemoveContainer" containerID="e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc"
May 26 12:30:46.787912 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.787877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 12:30:46.795050 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.795028    2563 scope.go:117] "RemoveContainer" containerID="d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a"
May 26 12:30:46.802068 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.802053    2563 scope.go:117] "RemoveContainer" containerID="8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46"
May 26 12:30:46.807941 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.807911    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"]
May 26 12:30:46.809639 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.809620    2563 scope.go:117] "RemoveContainer" containerID="e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc"
May 26 12:30:46.809960 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:30:46.809934    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc\": container with ID starting with e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc not found: ID does not exist" containerID="e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc"
May 26 12:30:46.810040 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.809971    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc"} err="failed to get container status \"e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc\": rpc error: code = NotFound desc = could not find container \"e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc\": container with ID starting with e8d87a718444f67653303c64cabefa8f240e96407a5da16131d34311ec4f93fc not found: ID does not exist"
May 26 12:30:46.810040 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.809990    2563 scope.go:117] "RemoveContainer" containerID="d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a"
May 26 12:30:46.810232 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:30:46.810210    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a\": container with ID starting with d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a not found: ID does not exist" containerID="d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a"
May 26 12:30:46.810306 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.810248    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a"} err="failed to get container status \"d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a\": rpc error: code = NotFound desc = could not find container \"d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a\": container with ID starting with d0ad54e010ec75afbc43ae3992fc5139db19ccd76f4c77f5ad0f4311c317dd7a not found: ID does not exist"
May 26 12:30:46.810306 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.810273    2563 scope.go:117] "RemoveContainer" containerID="8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46"
May 26 12:30:46.810588 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:30:46.810568    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46\": container with ID starting with 8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46 not found: ID does not exist" containerID="8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46"
May 26 12:30:46.810725 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.810595    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46"} err="failed to get container status \"8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46\": rpc error: code = NotFound desc = could not find container \"8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46\": container with ID starting with 8c83d8e2282bbbfb6e6fe6e1a6c945f88f233bea32ff5089e69034b90f15bf46 not found: ID does not exist"
May 26 12:30:46.811891 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:46.811872    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-44nnm"]
May 26 12:30:48.242053 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:48.242013    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" path="/var/lib/kubelet/pods/eeb8b40a-8a5b-42d4-9c0a-cc706de13de0/volumes"
May 26 12:30:51.791403 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:51.791371    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:30:51.791976 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:30:51.791951    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 12:31:01.792375 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:01.792338    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 12:31:11.792055 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:11.792016    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 12:31:21.792759 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:21.792716    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 12:31:31.792251 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:31.792212    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 12:31:41.792705 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:41.792632    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:31:48.972194 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:48.972164    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"]
May 26 12:31:48.972710 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:48.972459    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" containerID="cri-o://9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651" gracePeriod=30
May 26 12:31:48.972710 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:48.972539    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kube-rbac-proxy" containerID="cri-o://a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844" gracePeriod=30
May 26 12:31:49.048130 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048096    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"]
May 26 12:31:49.048376 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048363    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="storage-initializer"
May 26 12:31:49.048430 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048377    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="storage-initializer"
May 26 12:31:49.048430 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048386    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kube-rbac-proxy"
May 26 12:31:49.048430 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048393    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kube-rbac-proxy"
May 26 12:31:49.048430 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kserve-container"
May 26 12:31:49.048430 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048417    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kserve-container"
May 26 12:31:49.048615 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048456    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kube-rbac-proxy"
May 26 12:31:49.048615 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.048466    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="eeb8b40a-8a5b-42d4-9c0a-cc706de13de0" containerName="kserve-container"
May 26 12:31:49.051609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.051594    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.054176 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.054153    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
May 26 12:31:49.054365 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.054348    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 26 12:31:49.054449 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.054378    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
May 26 12:31:49.066116 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.066090    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"]
May 26 12:31:49.143560 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.143526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f29n7\" (UniqueName: \"kubernetes.io/projected/28f797f2-6617-49ff-b168-96982becb1b2-kube-api-access-f29n7\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.143717 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.143571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28f797f2-6617-49ff-b168-96982becb1b2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.143717 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.143635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28f797f2-6617-49ff-b168-96982becb1b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.143717 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.143681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28f797f2-6617-49ff-b168-96982becb1b2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.244418 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.244320    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28f797f2-6617-49ff-b168-96982becb1b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.244418 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.244369    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28f797f2-6617-49ff-b168-96982becb1b2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.244418 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.244408    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f29n7\" (UniqueName: \"kubernetes.io/projected/28f797f2-6617-49ff-b168-96982becb1b2-kube-api-access-f29n7\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.244698 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.244447    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28f797f2-6617-49ff-b168-96982becb1b2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.244878 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.244857    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28f797f2-6617-49ff-b168-96982becb1b2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.245118 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.245092    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28f797f2-6617-49ff-b168-96982becb1b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.247000 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.246985    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28f797f2-6617-49ff-b168-96982becb1b2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.253584 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.253559    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f29n7\" (UniqueName: \"kubernetes.io/projected/28f797f2-6617-49ff-b168-96982becb1b2-kube-api-access-f29n7\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.362068 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.362035    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:49.526231 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.526203    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"]
May 26 12:31:49.528922 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:31:49.528896    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f797f2_6617_49ff_b168_96982becb1b2.slice/crio-470793d4ff3f21b352424645a7f3b58e58adb7bff510ef678015ff1b2c77922f WatchSource:0}: Error finding container 470793d4ff3f21b352424645a7f3b58e58adb7bff510ef678015ff1b2c77922f: Status 404 returned error can't find the container with id 470793d4ff3f21b352424645a7f3b58e58adb7bff510ef678015ff1b2c77922f
May 26 12:31:49.954733 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.954691    2563 generic.go:358] "Generic (PLEG): container finished" podID="7550d808-292d-4897-b71e-db04dada687b" containerID="a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844" exitCode=2
May 26 12:31:49.954925 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.954760    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" event={"ID":"7550d808-292d-4897-b71e-db04dada687b","Type":"ContainerDied","Data":"a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844"}
May 26 12:31:49.956127 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.956103    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" event={"ID":"28f797f2-6617-49ff-b168-96982becb1b2","Type":"ContainerStarted","Data":"b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e"}
May 26 12:31:49.956127 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:49.956131    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" event={"ID":"28f797f2-6617-49ff-b168-96982becb1b2","Type":"ContainerStarted","Data":"470793d4ff3f21b352424645a7f3b58e58adb7bff510ef678015ff1b2c77922f"}
May 26 12:31:50.959914 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:50.959881    2563 generic.go:358] "Generic (PLEG): container finished" podID="28f797f2-6617-49ff-b168-96982becb1b2" containerID="b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e" exitCode=0
May 26 12:31:50.959914 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:50.959919    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" event={"ID":"28f797f2-6617-49ff-b168-96982becb1b2","Type":"ContainerDied","Data":"b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e"}
May 26 12:31:51.787811 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:51.787755    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
May 26 12:31:51.792029 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:51.791997    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 12:31:51.964071 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:51.964035    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" event={"ID":"28f797f2-6617-49ff-b168-96982becb1b2","Type":"ContainerStarted","Data":"d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926"}
May 26 12:31:51.964071 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:51.964073    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" event={"ID":"28f797f2-6617-49ff-b168-96982becb1b2","Type":"ContainerStarted","Data":"94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24"}
May 26 12:31:51.964527 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:51.964189    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:51.982886 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:51.982839    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podStartSLOduration=2.982823095 podStartE2EDuration="2.982823095s" podCreationTimestamp="2026-05-26 12:31:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:31:51.982155528 +0000 UTC m=+3874.345105197" watchObservedRunningTime="2026-05-26 12:31:51.982823095 +0000 UTC m=+3874.345772758"
May 26 12:31:52.709392 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.709370    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:31:52.770492 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.770413    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7550d808-292d-4897-b71e-db04dada687b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"7550d808-292d-4897-b71e-db04dada687b\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") "
May 26 12:31:52.770492 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.770464    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls\") pod \"7550d808-292d-4897-b71e-db04dada687b\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") "
May 26 12:31:52.770658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.770497    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wbh6\" (UniqueName: \"kubernetes.io/projected/7550d808-292d-4897-b71e-db04dada687b-kube-api-access-8wbh6\") pod \"7550d808-292d-4897-b71e-db04dada687b\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") "
May 26 12:31:52.770658 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.770560    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7550d808-292d-4897-b71e-db04dada687b-kserve-provision-location\") pod \"7550d808-292d-4897-b71e-db04dada687b\" (UID: \"7550d808-292d-4897-b71e-db04dada687b\") "
May 26 12:31:52.770898 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.770874    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7550d808-292d-4897-b71e-db04dada687b-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "7550d808-292d-4897-b71e-db04dada687b" (UID: "7550d808-292d-4897-b71e-db04dada687b"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:31:52.770963 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.770934    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7550d808-292d-4897-b71e-db04dada687b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7550d808-292d-4897-b71e-db04dada687b" (UID: "7550d808-292d-4897-b71e-db04dada687b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:31:52.772759 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.772742    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7550d808-292d-4897-b71e-db04dada687b-kube-api-access-8wbh6" (OuterVolumeSpecName: "kube-api-access-8wbh6") pod "7550d808-292d-4897-b71e-db04dada687b" (UID: "7550d808-292d-4897-b71e-db04dada687b"). InnerVolumeSpecName "kube-api-access-8wbh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:31:52.772902 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.772877    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7550d808-292d-4897-b71e-db04dada687b" (UID: "7550d808-292d-4897-b71e-db04dada687b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:31:52.872016 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.871973    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7550d808-292d-4897-b71e-db04dada687b-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:31:52.872016 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.872012    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7550d808-292d-4897-b71e-db04dada687b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:31:52.872016 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.872022    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7550d808-292d-4897-b71e-db04dada687b-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:31:52.872247 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.872034    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbh6\" (UniqueName: \"kubernetes.io/projected/7550d808-292d-4897-b71e-db04dada687b-kube-api-access-8wbh6\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:31:52.968180 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.968145    2563 generic.go:358] "Generic (PLEG): container finished" podID="7550d808-292d-4897-b71e-db04dada687b" containerID="9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651" exitCode=0
May 26 12:31:52.968558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.968214    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"
May 26 12:31:52.968558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.968224    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" event={"ID":"7550d808-292d-4897-b71e-db04dada687b","Type":"ContainerDied","Data":"9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651"}
May 26 12:31:52.968558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.968262    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r" event={"ID":"7550d808-292d-4897-b71e-db04dada687b","Type":"ContainerDied","Data":"603e0315aa644ce8c207e3c9eb5ba334ed0d228701e6d10cbf59995e53618d68"}
May 26 12:31:52.968558 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.968281    2563 scope.go:117] "RemoveContainer" containerID="a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844"
May 26 12:31:52.968848 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.968827    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:52.970209 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.970185    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 12:31:52.976549 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.976534    2563 scope.go:117] "RemoveContainer" containerID="9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651"
May 26 12:31:52.983402 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.983384    2563 scope.go:117] "RemoveContainer" containerID="f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf"
May 26 12:31:52.989959 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.989911    2563 scope.go:117] "RemoveContainer" containerID="a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844"
May 26 12:31:52.990299 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:31:52.990275    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844\": container with ID starting with a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844 not found: ID does not exist" containerID="a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844"
May 26 12:31:52.990379 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.990310    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844"} err="failed to get container status \"a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844\": rpc error: code = NotFound desc = could not find container \"a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844\": container with ID starting with a95991cea8938f706eee140b36aad40330c4f4252f6864eaa10c0510c0925844 not found: ID does not exist"
May 26 12:31:52.990379 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.990332    2563 scope.go:117] "RemoveContainer" containerID="9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651"
May 26 12:31:52.990563 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:31:52.990546    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651\": container with ID starting with 9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651 not found: ID does not exist" containerID="9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651"
May 26 12:31:52.990604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.990571    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651"} err="failed to get container status \"9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651\": rpc error: code = NotFound desc = could not find container \"9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651\": container with ID starting with 9cb5a0d178a47b731d5e8174c42f1052c393acab2c0b546a4d2eda66c63b9651 not found: ID does not exist"
May 26 12:31:52.990604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.990589    2563 scope.go:117] "RemoveContainer" containerID="f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf"
May 26 12:31:52.990837 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:31:52.990813    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf\": container with ID starting with f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf not found: ID does not exist" containerID="f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf"
May 26 12:31:52.990930 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.990841    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf"} err="failed to get container status \"f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf\": rpc error: code = NotFound desc = could not find container \"f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf\": container with ID starting with f2695731a564c32ce98e75f697afd405823f145defb331aca5e695907c801bdf not found: ID does not exist"
May 26 12:31:52.991491 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.991474    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"]
May 26 12:31:52.995377 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:52.995357    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-9jq2r"]
May 26 12:31:53.971741 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:53.971704    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 12:31:54.246521 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:54.243681    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7550d808-292d-4897-b71e-db04dada687b" path="/var/lib/kubelet/pods/7550d808-292d-4897-b71e-db04dada687b/volumes"
May 26 12:31:58.975837 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:58.975781    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:31:58.976336 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:31:58.976309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 12:32:08.977024 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:32:08.976980    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 12:32:18.976710 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:32:18.976668    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 12:32:28.977043 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:32:28.977005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 12:32:38.977105 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:32:38.977063    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 12:32:48.976948 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:32:48.976906    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 12:32:58.976997 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:32:58.976959    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:33:01.498940 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:01.498910    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:33:01.504477 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:01.504452    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:33:09.185665 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.185581    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"]
May 26 12:33:09.186166 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.185979    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container" containerID="cri-o://94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24" gracePeriod=30
May 26 12:33:09.186166 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.186020    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kube-rbac-proxy" containerID="cri-o://d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926" gracePeriod=30
May 26 12:33:09.325714 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.325676    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"]
May 26 12:33:09.325978 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.325964    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container"
May 26 12:33:09.326024 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.325980    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container"
May 26 12:33:09.326024 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.326000    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="storage-initializer"
May 26 12:33:09.326024 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.326007    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="storage-initializer"
May 26 12:33:09.326024 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.326017    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kube-rbac-proxy"
May 26 12:33:09.326024 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.326022    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kube-rbac-proxy"
May 26 12:33:09.326180 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.326064    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kube-rbac-proxy"
May 26 12:33:09.326180 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.326073    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7550d808-292d-4897-b71e-db04dada687b" containerName="kserve-container"
May 26 12:33:09.327882 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.327866    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.333414 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.333384    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
May 26 12:33:09.333414 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.333406    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 12:33:09.333657 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.333406    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
May 26 12:33:09.338735 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.338708    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"]
May 26 12:33:09.404865 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.404823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9tfd\" (UniqueName: \"kubernetes.io/projected/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kube-api-access-m9tfd\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.405086 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.404878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.405086 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.404903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.405086 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.404979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d491d4b-15c1-49a4-b268-7ab0845e2efa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.405086 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.405024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.506527 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.506427    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.506527 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.506495    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.506776 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.506534    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d491d4b-15c1-49a4-b268-7ab0845e2efa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.506776 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.506558    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.506776 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.506624    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tfd\" (UniqueName: \"kubernetes.io/projected/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kube-api-access-m9tfd\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.506991 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.506965    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.507274 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.507248    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.507350 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.507277    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.509273 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.509257    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d491d4b-15c1-49a4-b268-7ab0845e2efa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.514969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.514939    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tfd\" (UniqueName: \"kubernetes.io/projected/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kube-api-access-m9tfd\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.638569 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.638512    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:09.768700 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:09.768618    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"]
May 26 12:33:09.771887 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:33:09.771846    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d491d4b_15c1_49a4_b268_7ab0845e2efa.slice/crio-fd3b68b3385497bd45da5fc8d55d59b0d1a89db32eafdaaf942025458e75cdea WatchSource:0}: Error finding container fd3b68b3385497bd45da5fc8d55d59b0d1a89db32eafdaaf942025458e75cdea: Status 404 returned error can't find the container with id fd3b68b3385497bd45da5fc8d55d59b0d1a89db32eafdaaf942025458e75cdea
May 26 12:33:10.181286 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:10.181251    2563 generic.go:358] "Generic (PLEG): container finished" podID="28f797f2-6617-49ff-b168-96982becb1b2" containerID="d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926" exitCode=2
May 26 12:33:10.181512 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:10.181320    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" event={"ID":"28f797f2-6617-49ff-b168-96982becb1b2","Type":"ContainerDied","Data":"d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926"}
May 26 12:33:10.182710 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:10.182677    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" event={"ID":"7d491d4b-15c1-49a4-b268-7ab0845e2efa","Type":"ContainerStarted","Data":"9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721"}
May 26 12:33:10.182882 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:10.182715    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" event={"ID":"7d491d4b-15c1-49a4-b268-7ab0845e2efa","Type":"ContainerStarted","Data":"fd3b68b3385497bd45da5fc8d55d59b0d1a89db32eafdaaf942025458e75cdea"}
May 26 12:33:11.187211 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:11.187116    2563 generic.go:358] "Generic (PLEG): container finished" podID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerID="9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721" exitCode=0
May 26 12:33:11.187557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:11.187203    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" event={"ID":"7d491d4b-15c1-49a4-b268-7ab0845e2efa","Type":"ContainerDied","Data":"9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721"}
May 26 12:33:12.191491 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:12.191453    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" event={"ID":"7d491d4b-15c1-49a4-b268-7ab0845e2efa","Type":"ContainerStarted","Data":"6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94"}
May 26 12:33:12.191887 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:12.191500    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" event={"ID":"7d491d4b-15c1-49a4-b268-7ab0845e2efa","Type":"ContainerStarted","Data":"23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39"}
May 26 12:33:12.191887 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:12.191595    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:12.217593 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:12.217542    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podStartSLOduration=3.217526257 podStartE2EDuration="3.217526257s" podCreationTimestamp="2026-05-26 12:33:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:33:12.217252618 +0000 UTC m=+3954.580202288" watchObservedRunningTime="2026-05-26 12:33:12.217526257 +0000 UTC m=+3954.580475926"
May 26 12:33:13.194893 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.194858    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:13.196206 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.196177    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 12:33:13.733897 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.733874    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:33:13.842986 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.842958    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28f797f2-6617-49ff-b168-96982becb1b2-proxy-tls\") pod \"28f797f2-6617-49ff-b168-96982becb1b2\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") "
May 26 12:33:13.843170 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.842995    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28f797f2-6617-49ff-b168-96982becb1b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"28f797f2-6617-49ff-b168-96982becb1b2\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") "
May 26 12:33:13.843170 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.843015    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f29n7\" (UniqueName: \"kubernetes.io/projected/28f797f2-6617-49ff-b168-96982becb1b2-kube-api-access-f29n7\") pod \"28f797f2-6617-49ff-b168-96982becb1b2\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") "
May 26 12:33:13.843170 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.843050    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28f797f2-6617-49ff-b168-96982becb1b2-kserve-provision-location\") pod \"28f797f2-6617-49ff-b168-96982becb1b2\" (UID: \"28f797f2-6617-49ff-b168-96982becb1b2\") "
May 26 12:33:13.843387 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.843359    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f797f2-6617-49ff-b168-96982becb1b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28f797f2-6617-49ff-b168-96982becb1b2" (UID: "28f797f2-6617-49ff-b168-96982becb1b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:33:13.843453 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.843407    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f797f2-6617-49ff-b168-96982becb1b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "28f797f2-6617-49ff-b168-96982becb1b2" (UID: "28f797f2-6617-49ff-b168-96982becb1b2"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:33:13.845305 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.845290    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f797f2-6617-49ff-b168-96982becb1b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28f797f2-6617-49ff-b168-96982becb1b2" (UID: "28f797f2-6617-49ff-b168-96982becb1b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:33:13.845404 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.845385    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f797f2-6617-49ff-b168-96982becb1b2-kube-api-access-f29n7" (OuterVolumeSpecName: "kube-api-access-f29n7") pod "28f797f2-6617-49ff-b168-96982becb1b2" (UID: "28f797f2-6617-49ff-b168-96982becb1b2"). InnerVolumeSpecName "kube-api-access-f29n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:33:13.943576 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.943535    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28f797f2-6617-49ff-b168-96982becb1b2-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:33:13.943576 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.943569    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28f797f2-6617-49ff-b168-96982becb1b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:33:13.943576 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.943580    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f29n7\" (UniqueName: \"kubernetes.io/projected/28f797f2-6617-49ff-b168-96982becb1b2-kube-api-access-f29n7\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:33:13.943832 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:13.943591    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28f797f2-6617-49ff-b168-96982becb1b2-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:33:14.198472 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.198382    2563 generic.go:358] "Generic (PLEG): container finished" podID="28f797f2-6617-49ff-b168-96982becb1b2" containerID="94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24" exitCode=0
May 26 12:33:14.198877 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.198477    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"
May 26 12:33:14.198877 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.198474    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" event={"ID":"28f797f2-6617-49ff-b168-96982becb1b2","Type":"ContainerDied","Data":"94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24"}
May 26 12:33:14.198877 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.198517    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz" event={"ID":"28f797f2-6617-49ff-b168-96982becb1b2","Type":"ContainerDied","Data":"470793d4ff3f21b352424645a7f3b58e58adb7bff510ef678015ff1b2c77922f"}
May 26 12:33:14.198877 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.198537    2563 scope.go:117] "RemoveContainer" containerID="d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926"
May 26 12:33:14.199099 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.199064    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 12:33:14.206308 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.206287    2563 scope.go:117] "RemoveContainer" containerID="94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24"
May 26 12:33:14.213962 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.213945    2563 scope.go:117] "RemoveContainer" containerID="b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e"
May 26 12:33:14.220519 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.220501    2563 scope.go:117] "RemoveContainer" containerID="d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926"
May 26 12:33:14.220773 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:33:14.220754    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926\": container with ID starting with d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926 not found: ID does not exist" containerID="d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926"
May 26 12:33:14.220895 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.220783    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926"} err="failed to get container status \"d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926\": rpc error: code = NotFound desc = could not find container \"d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926\": container with ID starting with d76be2748d40dbf6ef4318d207596bb0ab971d72e3a931ff888656d603cbd926 not found: ID does not exist"
May 26 12:33:14.220941 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.220900    2563 scope.go:117] "RemoveContainer" containerID="94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24"
May 26 12:33:14.221185 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:33:14.221168    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24\": container with ID starting with 94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24 not found: ID does not exist" containerID="94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24"
May 26 12:33:14.221258 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.221194    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24"} err="failed to get container status \"94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24\": rpc error: code = NotFound desc = could not find container \"94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24\": container with ID starting with 94cf73c9fdfbf3c2cd31d73f8c5aff8156bf66d9e74d792fb3675d682afbda24 not found: ID does not exist"
May 26 12:33:14.221258 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.221216    2563 scope.go:117] "RemoveContainer" containerID="b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e"
May 26 12:33:14.221594 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:33:14.221568    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e\": container with ID starting with b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e not found: ID does not exist" containerID="b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e"
May 26 12:33:14.221685 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.221602    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e"} err="failed to get container status \"b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e\": rpc error: code = NotFound desc = could not find container \"b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e\": container with ID starting with b5b8c9fdaeaca5f8c33079d9c30836c5a43a85dc854c9ef8bc6c8878b2636b1e not found: ID does not exist"
May 26 12:33:14.222931 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.222914    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"]
May 26 12:33:14.227492 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.227471    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-dh2nz"]
May 26 12:33:14.242540 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:14.242508    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28f797f2-6617-49ff-b168-96982becb1b2" path="/var/lib/kubelet/pods/28f797f2-6617-49ff-b168-96982becb1b2/volumes"
May 26 12:33:19.203419 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:19.203388    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:33:19.204020 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:19.203986    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 12:33:29.204506 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:29.204462    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 12:33:39.204690 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:39.204645    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 12:33:49.204578 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:49.204534    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 12:33:59.204588 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:33:59.204545    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 12:34:09.204006 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:09.203948    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 12:34:19.204543 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:19.204501    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:34:29.333360 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:29.333330    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"]
May 26 12:34:29.333886 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:29.333663    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container" containerID="cri-o://23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39" gracePeriod=30
May 26 12:34:29.333886 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:29.333703    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kube-rbac-proxy" containerID="cri-o://6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94" gracePeriod=30
May 26 12:34:30.402308 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.402269    2563 generic.go:358] "Generic (PLEG): container finished" podID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerID="6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94" exitCode=2
May 26 12:34:30.402686 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.402313    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" event={"ID":"7d491d4b-15c1-49a4-b268-7ab0845e2efa","Type":"ContainerDied","Data":"6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94"}
May 26 12:34:30.444911 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.444874    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"]
May 26 12:34:30.445134 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.445122    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="storage-initializer"
May 26 12:34:30.445191 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.445135    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="storage-initializer"
May 26 12:34:30.445191 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.445143    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container"
May 26 12:34:30.445191 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.445149    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container"
May 26 12:34:30.445191 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.445161    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kube-rbac-proxy"
May 26 12:34:30.445191 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.445166    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kube-rbac-proxy"
May 26 12:34:30.445373 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.445209    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kserve-container"
May 26 12:34:30.445373 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.445217    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="28f797f2-6617-49ff-b168-96982becb1b2" containerName="kube-rbac-proxy"
May 26 12:34:30.446989 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.446972    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.448784 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.448739    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
May 26 12:34:30.448784 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.448766    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
May 26 12:34:30.460501 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.460477    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"]
May 26 12:34:30.523498 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.523466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.523498 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.523504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.523718 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.523525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf9n\" (UniqueName: \"kubernetes.io/projected/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kube-api-access-2qf9n\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.523718 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.523595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.624342 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.624303    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.624554 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.624352    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.624554 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.624380    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf9n\" (UniqueName: \"kubernetes.io/projected/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kube-api-access-2qf9n\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.624554 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.624420    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.624786 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.624760    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.625098 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.625076    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.626982 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.626965    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.633425 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.633403    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf9n\" (UniqueName: \"kubernetes.io/projected/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kube-api-access-2qf9n\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.758247 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.758157    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:30.888854 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:30.888821    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"]
May 26 12:34:31.407022 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:31.406980    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh" event={"ID":"e0ce28f1-8ebb-4a26-9b14-18be958ffe81","Type":"ContainerStarted","Data":"b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a"}
May 26 12:34:31.407022 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:31.407026    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh" event={"ID":"e0ce28f1-8ebb-4a26-9b14-18be958ffe81","Type":"ContainerStarted","Data":"15f4ed3a539f6a47bd714d4b75a9d789460a047e3606b4319e10a12a21458373"}
May 26 12:34:33.675967 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.675942    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:34:33.743052 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.743023    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kserve-provision-location\") pod \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") "
May 26 12:34:33.743251 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.743068    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") "
May 26 12:34:33.743251 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.743091    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9tfd\" (UniqueName: \"kubernetes.io/projected/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kube-api-access-m9tfd\") pod \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") "
May 26 12:34:33.743251 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.743106    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-cabundle-cert\") pod \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") "
May 26 12:34:33.743251 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.743228    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d491d4b-15c1-49a4-b268-7ab0845e2efa-proxy-tls\") pod \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\" (UID: \"7d491d4b-15c1-49a4-b268-7ab0845e2efa\") "
May 26 12:34:33.743487 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.743427    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d491d4b-15c1-49a4-b268-7ab0845e2efa" (UID: "7d491d4b-15c1-49a4-b268-7ab0845e2efa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:34:33.743540 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.743493    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "7d491d4b-15c1-49a4-b268-7ab0845e2efa" (UID: "7d491d4b-15c1-49a4-b268-7ab0845e2efa"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:34:33.743540 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.743500    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d491d4b-15c1-49a4-b268-7ab0845e2efa" (UID: "7d491d4b-15c1-49a4-b268-7ab0845e2efa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:34:33.745361 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.745339    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kube-api-access-m9tfd" (OuterVolumeSpecName: "kube-api-access-m9tfd") pod "7d491d4b-15c1-49a4-b268-7ab0845e2efa" (UID: "7d491d4b-15c1-49a4-b268-7ab0845e2efa"). InnerVolumeSpecName "kube-api-access-m9tfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:34:33.745467 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.745366    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d491d4b-15c1-49a4-b268-7ab0845e2efa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d491d4b-15c1-49a4-b268-7ab0845e2efa" (UID: "7d491d4b-15c1-49a4-b268-7ab0845e2efa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:34:33.844532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.844499    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d491d4b-15c1-49a4-b268-7ab0845e2efa-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:33.844532 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.844530    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:33.844740 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.844541    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:33.844740 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.844554    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9tfd\" (UniqueName: \"kubernetes.io/projected/7d491d4b-15c1-49a4-b268-7ab0845e2efa-kube-api-access-m9tfd\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:33.844740 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:33.844563    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d491d4b-15c1-49a4-b268-7ab0845e2efa-cabundle-cert\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:34.416924 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.416880    2563 generic.go:358] "Generic (PLEG): container finished" podID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerID="23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39" exitCode=0
May 26 12:34:34.417131 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.416938    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" event={"ID":"7d491d4b-15c1-49a4-b268-7ab0845e2efa","Type":"ContainerDied","Data":"23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39"}
May 26 12:34:34.417131 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.416959    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"
May 26 12:34:34.417131 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.416986    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg" event={"ID":"7d491d4b-15c1-49a4-b268-7ab0845e2efa","Type":"ContainerDied","Data":"fd3b68b3385497bd45da5fc8d55d59b0d1a89db32eafdaaf942025458e75cdea"}
May 26 12:34:34.417131 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.417008    2563 scope.go:117] "RemoveContainer" containerID="6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94"
May 26 12:34:34.425110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.425081    2563 scope.go:117] "RemoveContainer" containerID="23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39"
May 26 12:34:34.432316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.432269    2563 scope.go:117] "RemoveContainer" containerID="9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721"
May 26 12:34:34.433900 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.433876    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"]
May 26 12:34:34.439557 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.439532    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-4zczg"]
May 26 12:34:34.440985 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.440970    2563 scope.go:117] "RemoveContainer" containerID="6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94"
May 26 12:34:34.441288 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:34:34.441263    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94\": container with ID starting with 6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94 not found: ID does not exist" containerID="6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94"
May 26 12:34:34.441343 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.441291    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94"} err="failed to get container status \"6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94\": rpc error: code = NotFound desc = could not find container \"6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94\": container with ID starting with 6f0235385b2a5586b461a669c1860e932c05ea0b8e7e0020b6a7d78101bddf94 not found: ID does not exist"
May 26 12:34:34.441343 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.441312    2563 scope.go:117] "RemoveContainer" containerID="23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39"
May 26 12:34:34.441557 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:34:34.441541    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39\": container with ID starting with 23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39 not found: ID does not exist" containerID="23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39"
May 26 12:34:34.441613 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.441561    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39"} err="failed to get container status \"23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39\": rpc error: code = NotFound desc = could not find container \"23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39\": container with ID starting with 23250618538b153943942f61573ba88420b0cc83a0c4d170fc089a9c74988f39 not found: ID does not exist"
May 26 12:34:34.441613 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.441574    2563 scope.go:117] "RemoveContainer" containerID="9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721"
May 26 12:34:34.441823 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:34:34.441782    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721\": container with ID starting with 9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721 not found: ID does not exist" containerID="9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721"
May 26 12:34:34.441889 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:34.441834    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721"} err="failed to get container status \"9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721\": rpc error: code = NotFound desc = could not find container \"9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721\": container with ID starting with 9b2f1a07dbcbed30c9300b85c62dc364a05ad438874fb2f140b29ddf5a7c2721 not found: ID does not exist"
May 26 12:34:35.420921 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:35.420893    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh_e0ce28f1-8ebb-4a26-9b14-18be958ffe81/storage-initializer/0.log"
May 26 12:34:35.421369 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:35.420929    2563 generic.go:358] "Generic (PLEG): container finished" podID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerID="b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a" exitCode=1
May 26 12:34:35.421369 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:35.420953    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh" event={"ID":"e0ce28f1-8ebb-4a26-9b14-18be958ffe81","Type":"ContainerDied","Data":"b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a"}
May 26 12:34:36.242451 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:36.242420    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" path="/var/lib/kubelet/pods/7d491d4b-15c1-49a4-b268-7ab0845e2efa/volumes"
May 26 12:34:36.425168 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:36.425142    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh_e0ce28f1-8ebb-4a26-9b14-18be958ffe81/storage-initializer/0.log"
May 26 12:34:36.425545 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:36.425241    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh" event={"ID":"e0ce28f1-8ebb-4a26-9b14-18be958ffe81","Type":"ContainerStarted","Data":"b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921"}
May 26 12:34:40.412219 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:40.412103    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"]
May 26 12:34:40.412722 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:40.412399    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh" podUID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerName="storage-initializer" containerID="cri-o://b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921" gracePeriod=30
May 26 12:34:41.249085 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.249056    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh_e0ce28f1-8ebb-4a26-9b14-18be958ffe81/storage-initializer/1.log"
May 26 12:34:41.249462 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.249449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh_e0ce28f1-8ebb-4a26-9b14-18be958ffe81/storage-initializer/0.log"
May 26 12:34:41.249520 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.249510    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:41.301108 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.301074    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kserve-provision-location\") pod \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") "
May 26 12:34:41.301108 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.301107    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qf9n\" (UniqueName: \"kubernetes.io/projected/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kube-api-access-2qf9n\") pod \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") "
May 26 12:34:41.301326 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.301146    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") "
May 26 12:34:41.301326 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.301180    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-proxy-tls\") pod \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\" (UID: \"e0ce28f1-8ebb-4a26-9b14-18be958ffe81\") "
May 26 12:34:41.301463 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.301414    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0ce28f1-8ebb-4a26-9b14-18be958ffe81" (UID: "e0ce28f1-8ebb-4a26-9b14-18be958ffe81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:34:41.301570 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.301539    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "e0ce28f1-8ebb-4a26-9b14-18be958ffe81" (UID: "e0ce28f1-8ebb-4a26-9b14-18be958ffe81"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:34:41.303375 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.303343    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0ce28f1-8ebb-4a26-9b14-18be958ffe81" (UID: "e0ce28f1-8ebb-4a26-9b14-18be958ffe81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:34:41.303450 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.303432    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kube-api-access-2qf9n" (OuterVolumeSpecName: "kube-api-access-2qf9n") pod "e0ce28f1-8ebb-4a26-9b14-18be958ffe81" (UID: "e0ce28f1-8ebb-4a26-9b14-18be958ffe81"). InnerVolumeSpecName "kube-api-access-2qf9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:34:41.402046 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.401957    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:41.402046 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.401989    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qf9n\" (UniqueName: \"kubernetes.io/projected/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-kube-api-access-2qf9n\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:41.402046 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.402003    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:41.402046 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.402019    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0ce28f1-8ebb-4a26-9b14-18be958ffe81-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:34:41.439598 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.439568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh_e0ce28f1-8ebb-4a26-9b14-18be958ffe81/storage-initializer/1.log"
May 26 12:34:41.440026 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.439902    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh_e0ce28f1-8ebb-4a26-9b14-18be958ffe81/storage-initializer/0.log"
May 26 12:34:41.440026 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.439934    2563 generic.go:358] "Generic (PLEG): container finished" podID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerID="b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921" exitCode=1
May 26 12:34:41.440102 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.440032    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"
May 26 12:34:41.440141 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.440023    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh" event={"ID":"e0ce28f1-8ebb-4a26-9b14-18be958ffe81","Type":"ContainerDied","Data":"b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921"}
May 26 12:34:41.440177 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.440156    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh" event={"ID":"e0ce28f1-8ebb-4a26-9b14-18be958ffe81","Type":"ContainerDied","Data":"15f4ed3a539f6a47bd714d4b75a9d789460a047e3606b4319e10a12a21458373"}
May 26 12:34:41.440213 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.440178    2563 scope.go:117] "RemoveContainer" containerID="b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921"
May 26 12:34:41.449081 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.448859    2563 scope.go:117] "RemoveContainer" containerID="b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a"
May 26 12:34:41.456288 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.456270    2563 scope.go:117] "RemoveContainer" containerID="b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921"
May 26 12:34:41.456494 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:34:41.456477    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921\": container with ID starting with b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921 not found: ID does not exist" containerID="b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921"
May 26 12:34:41.456544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.456501    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921"} err="failed to get container status \"b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921\": rpc error: code = NotFound desc = could not find container \"b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921\": container with ID starting with b372f4c2ead9940a4d11777749a660dd6b6066900b7340fa59c6fde761d5f921 not found: ID does not exist"
May 26 12:34:41.456544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.456517    2563 scope.go:117] "RemoveContainer" containerID="b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a"
May 26 12:34:41.456777 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:34:41.456757    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a\": container with ID starting with b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a not found: ID does not exist" containerID="b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a"
May 26 12:34:41.456860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.456784    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a"} err="failed to get container status \"b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a\": rpc error: code = NotFound desc = could not find container \"b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a\": container with ID starting with b03f30aa36fc0180d1030aca91cc3f97921e619e3b27baa38319493a83f5ee0a not found: ID does not exist"
May 26 12:34:41.478357 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.478328    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"]
May 26 12:34:41.487975 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.487931    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xw2nh"]
May 26 12:34:41.494026 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.493996    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"]
May 26 12:34:41.494300 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container"
May 26 12:34:41.494395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494303    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container"
May 26 12:34:41.494395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494323    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="storage-initializer"
May 26 12:34:41.494395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494333    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="storage-initializer"
May 26 12:34:41.494395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494352    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kube-rbac-proxy"
May 26 12:34:41.494395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494361    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kube-rbac-proxy"
May 26 12:34:41.494395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494372    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerName="storage-initializer"
May 26 12:34:41.494395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494381    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerName="storage-initializer"
May 26 12:34:41.494395 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494391    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerName="storage-initializer"
May 26 12:34:41.494828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494400    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerName="storage-initializer"
May 26 12:34:41.494828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494515    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerName="storage-initializer"
May 26 12:34:41.494828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494531    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" containerName="storage-initializer"
May 26 12:34:41.494828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494542    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kserve-container"
May 26 12:34:41.494828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.494553    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d491d4b-15c1-49a4-b268-7ab0845e2efa" containerName="kube-rbac-proxy"
May 26 12:34:41.497290 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.497269    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.499308 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.499283    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
May 26 12:34:41.499428 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.499297    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 12:34:41.499428 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.499296    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b5sxz\""
May 26 12:34:41.499428 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.499297    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 26 12:34:41.499428 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.499304    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
May 26 12:34:41.499659 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.499345    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 12:34:41.499659 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.499406    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 12:34:41.506354 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.506330    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"]
May 26 12:34:41.603250 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.603215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.603250 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.603257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.603477 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.603275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78l8\" (UniqueName: \"kubernetes.io/projected/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kube-api-access-t78l8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.603477 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.603366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.603477 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.603407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.704691 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.704584    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.704691 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.704636    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.704691 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.704663    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.704691 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.704693    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.705084 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.704711    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t78l8\" (UniqueName: \"kubernetes.io/projected/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kube-api-access-t78l8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.705084 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:34:41.704851    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
May 26 12:34:41.705084 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:34:41.704924    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls podName:09a172bc-4fc9-4832-a200-5218b9ef9ea2 nodeName:}" failed. No retries permitted until 2026-05-26 12:34:42.204907115 +0000 UTC m=+4044.567856761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" (UID: "09a172bc-4fc9-4832-a200-5218b9ef9ea2") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
May 26 12:34:41.705246 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.705150    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.705416 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.705394    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.705476 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.705401    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:41.713329 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:41.713306    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78l8\" (UniqueName: \"kubernetes.io/projected/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kube-api-access-t78l8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:42.207537 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:42.207496    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:42.210151 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:42.210129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:42.242255 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:42.242224    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0ce28f1-8ebb-4a26-9b14-18be958ffe81" path="/var/lib/kubelet/pods/e0ce28f1-8ebb-4a26-9b14-18be958ffe81/volumes"
May 26 12:34:42.408470 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:42.408429    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:42.547105 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:42.547073    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"]
May 26 12:34:42.550264 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:34:42.550236    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a172bc_4fc9_4832_a200_5218b9ef9ea2.slice/crio-89431e3b7922057a34ab6c3eee63f68bdc053f0997f86ec78c7faa87736dceda WatchSource:0}: Error finding container 89431e3b7922057a34ab6c3eee63f68bdc053f0997f86ec78c7faa87736dceda: Status 404 returned error can't find the container with id 89431e3b7922057a34ab6c3eee63f68bdc053f0997f86ec78c7faa87736dceda
May 26 12:34:43.448071 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:43.448032    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" event={"ID":"09a172bc-4fc9-4832-a200-5218b9ef9ea2","Type":"ContainerStarted","Data":"ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218"}
May 26 12:34:43.448259 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:43.448078    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" event={"ID":"09a172bc-4fc9-4832-a200-5218b9ef9ea2","Type":"ContainerStarted","Data":"89431e3b7922057a34ab6c3eee63f68bdc053f0997f86ec78c7faa87736dceda"}
May 26 12:34:44.452183 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:44.452143    2563 generic.go:358] "Generic (PLEG): container finished" podID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerID="ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218" exitCode=0
May 26 12:34:44.452563 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:44.452205    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" event={"ID":"09a172bc-4fc9-4832-a200-5218b9ef9ea2","Type":"ContainerDied","Data":"ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218"}
May 26 12:34:45.459449 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:45.459413    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" event={"ID":"09a172bc-4fc9-4832-a200-5218b9ef9ea2","Type":"ContainerStarted","Data":"ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662"}
May 26 12:34:45.459449 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:45.459454    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" event={"ID":"09a172bc-4fc9-4832-a200-5218b9ef9ea2","Type":"ContainerStarted","Data":"2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20"}
May 26 12:34:45.459968 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:45.459625    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:45.459968 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:45.459733    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:45.461052 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:45.461022    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 12:34:45.478028 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:45.477980    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podStartSLOduration=4.477964732 podStartE2EDuration="4.477964732s" podCreationTimestamp="2026-05-26 12:34:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:34:45.476192956 +0000 UTC m=+4047.839142626" watchObservedRunningTime="2026-05-26 12:34:45.477964732 +0000 UTC m=+4047.840914400"
May 26 12:34:46.462933 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:46.462894    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 12:34:51.466969 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:51.466939    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:34:51.467517 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:34:51.467490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 12:35:01.468231 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:35:01.468182    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 12:35:11.467399 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:35:11.467357    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 12:35:21.467380 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:35:21.467339    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 12:35:31.467925 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:35:31.467882    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 12:35:41.467818 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:35:41.467764    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 12:35:51.468011 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:35:51.467981    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:36:01.513086 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:01.513053    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"]
May 26 12:36:01.513605 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:01.513363    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container" containerID="cri-o://2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20" gracePeriod=30
May 26 12:36:01.513605 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:01.513400    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kube-rbac-proxy" containerID="cri-o://ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662" gracePeriod=30
May 26 12:36:01.663941 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:01.663902    2563 generic.go:358] "Generic (PLEG): container finished" podID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerID="ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662" exitCode=2
May 26 12:36:01.664117 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:01.663972    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" event={"ID":"09a172bc-4fc9-4832-a200-5218b9ef9ea2","Type":"ContainerDied","Data":"ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662"}
May 26 12:36:02.582638 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.582598    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"]
May 26 12:36:02.586110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.586089    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.588085 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.588063    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
May 26 12:36:02.588211 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.588082    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
May 26 12:36:02.592425 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.592398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4353a87f-85c6-403c-8e1e-4f2bc030caf2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.592543 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.592437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrnq\" (UniqueName: \"kubernetes.io/projected/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kube-api-access-zmrnq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.592543 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.592512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.592666 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.592543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4353a87f-85c6-403c-8e1e-4f2bc030caf2-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.596367 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.596345    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"]
May 26 12:36:02.693720 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.693684    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.693720 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.693726    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4353a87f-85c6-403c-8e1e-4f2bc030caf2-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.694025 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.693752    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4353a87f-85c6-403c-8e1e-4f2bc030caf2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.694025 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.693785    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrnq\" (UniqueName: \"kubernetes.io/projected/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kube-api-access-zmrnq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.694147 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.694128    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.694405 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.694386    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4353a87f-85c6-403c-8e1e-4f2bc030caf2-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.696370 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.696352    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4353a87f-85c6-403c-8e1e-4f2bc030caf2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.701572 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.701550    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrnq\" (UniqueName: \"kubernetes.io/projected/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kube-api-access-zmrnq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:02.897578 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:02.897495    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:03.024006 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:03.023974    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"]
May 26 12:36:03.027239 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:36:03.027208    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4353a87f_85c6_403c_8e1e_4f2bc030caf2.slice/crio-f1e4d9063f27aff61bed967cf5d8f2dbc0bace882c5d0cb50afe171496fd393c WatchSource:0}: Error finding container f1e4d9063f27aff61bed967cf5d8f2dbc0bace882c5d0cb50afe171496fd393c: Status 404 returned error can't find the container with id f1e4d9063f27aff61bed967cf5d8f2dbc0bace882c5d0cb50afe171496fd393c
May 26 12:36:03.028943 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:03.028926    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:36:03.672204 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:03.672163    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x" event={"ID":"4353a87f-85c6-403c-8e1e-4f2bc030caf2","Type":"ContainerStarted","Data":"1be8275f909be416098abced9b89b653dccee2b607bb1effbf5538b94f557cfa"}
May 26 12:36:03.672204 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:03.672207    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x" event={"ID":"4353a87f-85c6-403c-8e1e-4f2bc030caf2","Type":"ContainerStarted","Data":"f1e4d9063f27aff61bed967cf5d8f2dbc0bace882c5d0cb50afe171496fd393c"}
May 26 12:36:05.880032 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.880009    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:36:05.918852 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.918774    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") "
May 26 12:36:05.918852 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.918825    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-cabundle-cert\") pod \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") "
May 26 12:36:05.918852 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.918850    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t78l8\" (UniqueName: \"kubernetes.io/projected/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kube-api-access-t78l8\") pod \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") "
May 26 12:36:05.919121 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.918868    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kserve-provision-location\") pod \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") "
May 26 12:36:05.919121 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.918887    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls\") pod \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\" (UID: \"09a172bc-4fc9-4832-a200-5218b9ef9ea2\") "
May 26 12:36:05.919272 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.919244    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09a172bc-4fc9-4832-a200-5218b9ef9ea2" (UID: "09a172bc-4fc9-4832-a200-5218b9ef9ea2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:36:05.919272 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.919258    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "09a172bc-4fc9-4832-a200-5218b9ef9ea2" (UID: "09a172bc-4fc9-4832-a200-5218b9ef9ea2"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:36:05.919372 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.919280    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09a172bc-4fc9-4832-a200-5218b9ef9ea2" (UID: "09a172bc-4fc9-4832-a200-5218b9ef9ea2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:36:05.921021 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.921002    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kube-api-access-t78l8" (OuterVolumeSpecName: "kube-api-access-t78l8") pod "09a172bc-4fc9-4832-a200-5218b9ef9ea2" (UID: "09a172bc-4fc9-4832-a200-5218b9ef9ea2"). InnerVolumeSpecName "kube-api-access-t78l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:36:05.921089 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:05.921053    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09a172bc-4fc9-4832-a200-5218b9ef9ea2" (UID: "09a172bc-4fc9-4832-a200-5218b9ef9ea2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:36:06.019683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.019582    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:06.019683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.019627    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a172bc-4fc9-4832-a200-5218b9ef9ea2-cabundle-cert\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:06.019683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.019637    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t78l8\" (UniqueName: \"kubernetes.io/projected/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kube-api-access-t78l8\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:06.019683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.019652    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a172bc-4fc9-4832-a200-5218b9ef9ea2-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:06.019683 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.019666    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a172bc-4fc9-4832-a200-5218b9ef9ea2-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:06.681892 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.681857    2563 generic.go:358] "Generic (PLEG): container finished" podID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerID="2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20" exitCode=0
May 26 12:36:06.682110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.681925    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" event={"ID":"09a172bc-4fc9-4832-a200-5218b9ef9ea2","Type":"ContainerDied","Data":"2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20"}
May 26 12:36:06.682110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.681943    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"
May 26 12:36:06.682110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.681953    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph" event={"ID":"09a172bc-4fc9-4832-a200-5218b9ef9ea2","Type":"ContainerDied","Data":"89431e3b7922057a34ab6c3eee63f68bdc053f0997f86ec78c7faa87736dceda"}
May 26 12:36:06.682110 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.681967    2563 scope.go:117] "RemoveContainer" containerID="ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662"
May 26 12:36:06.689725 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.689709    2563 scope.go:117] "RemoveContainer" containerID="2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20"
May 26 12:36:06.697345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.697329    2563 scope.go:117] "RemoveContainer" containerID="ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218"
May 26 12:36:06.699436 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.699414    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"]
May 26 12:36:06.704747 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.704725    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-r72ph"]
May 26 12:36:06.705127 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.705103    2563 scope.go:117] "RemoveContainer" containerID="ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662"
May 26 12:36:06.705364 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:36:06.705345    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662\": container with ID starting with ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662 not found: ID does not exist" containerID="ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662"
May 26 12:36:06.705414 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.705374    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662"} err="failed to get container status \"ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662\": rpc error: code = NotFound desc = could not find container \"ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662\": container with ID starting with ef764587e1251026ce2123881c253385e25b2edbafdd1ffdfeaf0f5a40f2c662 not found: ID does not exist"
May 26 12:36:06.705414 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.705393    2563 scope.go:117] "RemoveContainer" containerID="2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20"
May 26 12:36:06.705626 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:36:06.705608    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20\": container with ID starting with 2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20 not found: ID does not exist" containerID="2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20"
May 26 12:36:06.705666 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.705632    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20"} err="failed to get container status \"2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20\": rpc error: code = NotFound desc = could not find container \"2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20\": container with ID starting with 2512f21d7c129b70c884cc48ff0d3b10795f01fb657208544d953dd981345f20 not found: ID does not exist"
May 26 12:36:06.705666 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.705647    2563 scope.go:117] "RemoveContainer" containerID="ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218"
May 26 12:36:06.705887 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:36:06.705867    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218\": container with ID starting with ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218 not found: ID does not exist" containerID="ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218"
May 26 12:36:06.705972 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:06.705889    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218"} err="failed to get container status \"ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218\": rpc error: code = NotFound desc = could not find container \"ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218\": container with ID starting with ec436c15f8c223ffa93884b617b67ebe388c51794fbc86f7614d32a7de781218 not found: ID does not exist"
May 26 12:36:08.242104 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:08.242076    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" path="/var/lib/kubelet/pods/09a172bc-4fc9-4832-a200-5218b9ef9ea2/volumes"
May 26 12:36:08.689289 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:08.689260    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x_4353a87f-85c6-403c-8e1e-4f2bc030caf2/storage-initializer/0.log"
May 26 12:36:08.689468 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:08.689300    2563 generic.go:358] "Generic (PLEG): container finished" podID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerID="1be8275f909be416098abced9b89b653dccee2b607bb1effbf5538b94f557cfa" exitCode=1
May 26 12:36:08.689468 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:08.689379    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x" event={"ID":"4353a87f-85c6-403c-8e1e-4f2bc030caf2","Type":"ContainerDied","Data":"1be8275f909be416098abced9b89b653dccee2b607bb1effbf5538b94f557cfa"}
May 26 12:36:09.693280 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:09.693253    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x_4353a87f-85c6-403c-8e1e-4f2bc030caf2/storage-initializer/0.log"
May 26 12:36:09.693653 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:09.693334    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x" event={"ID":"4353a87f-85c6-403c-8e1e-4f2bc030caf2","Type":"ContainerStarted","Data":"bd3ca3f4cadf152c4d3ae234ad611d3c595bd7b71f301c7c970a24620d22984a"}
May 26 12:36:12.722784 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:12.722743    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"]
May 26 12:36:12.723259 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:12.723033    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x" podUID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerName="storage-initializer" containerID="cri-o://bd3ca3f4cadf152c4d3ae234ad611d3c595bd7b71f301c7c970a24620d22984a" gracePeriod=30
May 26 12:36:13.698928 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.698901    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"]
May 26 12:36:13.699172 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.699159    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kube-rbac-proxy"
May 26 12:36:13.699228 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.699173    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kube-rbac-proxy"
May 26 12:36:13.699228 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.699185    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="storage-initializer"
May 26 12:36:13.699228 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.699190    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="storage-initializer"
May 26 12:36:13.699228 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.699197    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container"
May 26 12:36:13.699228 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.699202    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container"
May 26 12:36:13.699371 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.699249    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kserve-container"
May 26 12:36:13.699371 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.699259    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="09a172bc-4fc9-4832-a200-5218b9ef9ea2" containerName="kube-rbac-proxy"
May 26 12:36:13.701360 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.701321    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.703717 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.703696    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
May 26 12:36:13.703988 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.703969    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
May 26 12:36:13.704136 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.704083    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 12:36:13.707829 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.707787    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x_4353a87f-85c6-403c-8e1e-4f2bc030caf2/storage-initializer/1.log"
May 26 12:36:13.708237 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.708221    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x_4353a87f-85c6-403c-8e1e-4f2bc030caf2/storage-initializer/0.log"
May 26 12:36:13.708316 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.708263    2563 generic.go:358] "Generic (PLEG): container finished" podID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerID="bd3ca3f4cadf152c4d3ae234ad611d3c595bd7b71f301c7c970a24620d22984a" exitCode=1
May 26 12:36:13.708373 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.708340    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x" event={"ID":"4353a87f-85c6-403c-8e1e-4f2bc030caf2","Type":"ContainerDied","Data":"bd3ca3f4cadf152c4d3ae234ad611d3c595bd7b71f301c7c970a24620d22984a"}
May 26 12:36:13.708426 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.708393    2563 scope.go:117] "RemoveContainer" containerID="1be8275f909be416098abced9b89b653dccee2b607bb1effbf5538b94f557cfa"
May 26 12:36:13.713916 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.713894    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"]
May 26 12:36:13.763699 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.763680    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x_4353a87f-85c6-403c-8e1e-4f2bc030caf2/storage-initializer/1.log"
May 26 12:36:13.764048 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.763741    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:13.775440 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775416    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmrnq\" (UniqueName: \"kubernetes.io/projected/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kube-api-access-zmrnq\") pod \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") "
May 26 12:36:13.775544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775452    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4353a87f-85c6-403c-8e1e-4f2bc030caf2-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") "
May 26 12:36:13.775544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775475    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kserve-provision-location\") pod \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") "
May 26 12:36:13.775544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775508    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4353a87f-85c6-403c-8e1e-4f2bc030caf2-proxy-tls\") pod \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\" (UID: \"4353a87f-85c6-403c-8e1e-4f2bc030caf2\") "
May 26 12:36:13.775680 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.775680 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.775776 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.775776 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775731    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4353a87f-85c6-403c-8e1e-4f2bc030caf2" (UID: "4353a87f-85c6-403c-8e1e-4f2bc030caf2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:36:13.775776 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkxx\" (UniqueName: \"kubernetes.io/projected/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kube-api-access-cmkxx\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.775929 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775824    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4353a87f-85c6-403c-8e1e-4f2bc030caf2-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "4353a87f-85c6-403c-8e1e-4f2bc030caf2" (UID: "4353a87f-85c6-403c-8e1e-4f2bc030caf2"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:36:13.775929 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.775929 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775918    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4353a87f-85c6-403c-8e1e-4f2bc030caf2-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:13.776036 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.775932    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:13.777764 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.777741    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4353a87f-85c6-403c-8e1e-4f2bc030caf2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4353a87f-85c6-403c-8e1e-4f2bc030caf2" (UID: "4353a87f-85c6-403c-8e1e-4f2bc030caf2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:36:13.777860 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.777786    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kube-api-access-zmrnq" (OuterVolumeSpecName: "kube-api-access-zmrnq") pod "4353a87f-85c6-403c-8e1e-4f2bc030caf2" (UID: "4353a87f-85c6-403c-8e1e-4f2bc030caf2"). InnerVolumeSpecName "kube-api-access-zmrnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:36:13.877035 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.876946    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.877035 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.876991    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.877263 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.877162    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.877263 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.877196    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkxx\" (UniqueName: \"kubernetes.io/projected/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kube-api-access-cmkxx\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.877263 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.877225    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.877263 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.877257    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmrnq\" (UniqueName: \"kubernetes.io/projected/4353a87f-85c6-403c-8e1e-4f2bc030caf2-kube-api-access-zmrnq\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:13.877460 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.877273    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4353a87f-85c6-403c-8e1e-4f2bc030caf2-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:36:13.877460 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.877415    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.877706 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.877686    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.877885 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.877868    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.879757 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.879734    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:13.885444 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:13.885425    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkxx\" (UniqueName: \"kubernetes.io/projected/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kube-api-access-cmkxx\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:14.015108 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.015068    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:14.141462 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.141377    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"]
May 26 12:36:14.144672 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:36:14.144634    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a7e7c5_a826_4bdf_83f1_2a0c0a835666.slice/crio-d0f152577f8826465992354ed8abc177125f9753f093ede28c25d70c13997db1 WatchSource:0}: Error finding container d0f152577f8826465992354ed8abc177125f9753f093ede28c25d70c13997db1: Status 404 returned error can't find the container with id d0f152577f8826465992354ed8abc177125f9753f093ede28c25d70c13997db1
May 26 12:36:14.714871 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.714845    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x_4353a87f-85c6-403c-8e1e-4f2bc030caf2/storage-initializer/1.log"
May 26 12:36:14.715042 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.714935    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x" event={"ID":"4353a87f-85c6-403c-8e1e-4f2bc030caf2","Type":"ContainerDied","Data":"f1e4d9063f27aff61bed967cf5d8f2dbc0bace882c5d0cb50afe171496fd393c"}
May 26 12:36:14.715042 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.714964    2563 scope.go:117] "RemoveContainer" containerID="bd3ca3f4cadf152c4d3ae234ad611d3c595bd7b71f301c7c970a24620d22984a"
May 26 12:36:14.715042 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.714979    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"
May 26 12:36:14.719323 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.717200    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" event={"ID":"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666","Type":"ContainerStarted","Data":"8250b343817629da8b7dcb4891a00f38eb1f0c68bca14a7d0fd4eeb3da77bf7e"}
May 26 12:36:14.719323 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.717270    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" event={"ID":"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666","Type":"ContainerStarted","Data":"d0f152577f8826465992354ed8abc177125f9753f093ede28c25d70c13997db1"}
May 26 12:36:14.762365 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.762334    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"]
May 26 12:36:14.766379 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:14.766354    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-nh66x"]
May 26 12:36:15.721255 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:15.721226    2563 generic.go:358] "Generic (PLEG): container finished" podID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerID="8250b343817629da8b7dcb4891a00f38eb1f0c68bca14a7d0fd4eeb3da77bf7e" exitCode=0
May 26 12:36:15.721407 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:15.721276    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" event={"ID":"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666","Type":"ContainerDied","Data":"8250b343817629da8b7dcb4891a00f38eb1f0c68bca14a7d0fd4eeb3da77bf7e"}
May 26 12:36:16.242497 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:16.242457    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" path="/var/lib/kubelet/pods/4353a87f-85c6-403c-8e1e-4f2bc030caf2/volumes"
May 26 12:36:16.725555 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:16.725519    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" event={"ID":"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666","Type":"ContainerStarted","Data":"9507536ed118442060063ba82f5e699972c940fb4168f93502f1f464f279791e"}
May 26 12:36:16.725733 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:16.725562    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" event={"ID":"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666","Type":"ContainerStarted","Data":"4c4dac77941f6cfd07497f14dad2e04228762cf08c6b53badb0216faee60772a"}
May 26 12:36:16.725733 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:16.725719    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:16.750785 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:16.750737    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podStartSLOduration=3.750723853 podStartE2EDuration="3.750723853s" podCreationTimestamp="2026-05-26 12:36:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:36:16.748989737 +0000 UTC m=+4139.111939420" watchObservedRunningTime="2026-05-26 12:36:16.750723853 +0000 UTC m=+4139.113673523"
May 26 12:36:17.728142 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:17.728103    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:17.728955 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:17.728931    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 12:36:18.730696 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:18.730650    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 12:36:23.737269 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:23.737241    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:36:23.737806 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:23.737765    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 12:36:33.738319 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:33.738271    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 12:36:43.738035 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:43.737991    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 12:36:53.737983 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:36:53.737945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 12:37:03.738093 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:03.738053    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 12:37:13.738023 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:13.737977    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 12:37:23.739230 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:23.739193    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:37:33.732147 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:33.732109    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"]
May 26 12:37:33.732615 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:33.732532    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container" containerID="cri-o://4c4dac77941f6cfd07497f14dad2e04228762cf08c6b53badb0216faee60772a" gracePeriod=30
May 26 12:37:33.732686 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:33.732604    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kube-rbac-proxy" containerID="cri-o://9507536ed118442060063ba82f5e699972c940fb4168f93502f1f464f279791e" gracePeriod=30
May 26 12:37:33.931218 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:33.931187    2563 generic.go:358] "Generic (PLEG): container finished" podID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerID="9507536ed118442060063ba82f5e699972c940fb4168f93502f1f464f279791e" exitCode=2
May 26 12:37:33.931378 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:33.931224    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" event={"ID":"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666","Type":"ContainerDied","Data":"9507536ed118442060063ba82f5e699972c940fb4168f93502f1f464f279791e"}
May 26 12:37:34.769536 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.769503    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"]
May 26 12:37:34.769919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.769758    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerName="storage-initializer"
May 26 12:37:34.769919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.769770    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerName="storage-initializer"
May 26 12:37:34.769919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.769784    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerName="storage-initializer"
May 26 12:37:34.769919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.769807    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerName="storage-initializer"
May 26 12:37:34.769919 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.769853    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerName="storage-initializer"
May 26 12:37:34.770083 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.769941    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="4353a87f-85c6-403c-8e1e-4f2bc030caf2" containerName="storage-initializer"
May 26 12:37:34.772764 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.772739    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:34.774854 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.774828    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
May 26 12:37:34.774955 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.774907    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
May 26 12:37:34.784345 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.784326    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"]
May 26 12:37:34.947370 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.947334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:34.947540 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.947397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:34.947540 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.947438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:34.947540 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:34.947455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8h7\" (UniqueName: \"kubernetes.io/projected/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kube-api-access-gg8h7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.048231 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.048195    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.048413 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.048241    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8h7\" (UniqueName: \"kubernetes.io/projected/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kube-api-access-gg8h7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.048413 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.048304    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.048413 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:37:35.048354    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
May 26 12:37:35.048604 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:37:35.048420    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls podName:fbabfa19-0d61-4afb-bd4e-79f80130ed1a nodeName:}" failed. No retries permitted until 2026-05-26 12:37:35.548398097 +0000 UTC m=+4217.911347744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t" (UID: "fbabfa19-0d61-4afb-bd4e-79f80130ed1a") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
May 26 12:37:35.048604 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.048355    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.048759 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.048740    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.049011 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.048991    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.057412 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.057391    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8h7\" (UniqueName: \"kubernetes.io/projected/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kube-api-access-gg8h7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.553014 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.552978    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.555599 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.555565    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.682204 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.682109    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:35.811039 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.810965    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"]
May 26 12:37:35.814123 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:37:35.814097    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbabfa19_0d61_4afb_bd4e_79f80130ed1a.slice/crio-8d444b6f534aaa5d606464fc4421ae6c534a5d9735699822211fce0cb683ae35 WatchSource:0}: Error finding container 8d444b6f534aaa5d606464fc4421ae6c534a5d9735699822211fce0cb683ae35: Status 404 returned error can't find the container with id 8d444b6f534aaa5d606464fc4421ae6c534a5d9735699822211fce0cb683ae35
May 26 12:37:35.938583 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.938551    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t" event={"ID":"fbabfa19-0d61-4afb-bd4e-79f80130ed1a","Type":"ContainerStarted","Data":"2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007"}
May 26 12:37:35.938583 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:35.938588    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t" event={"ID":"fbabfa19-0d61-4afb-bd4e-79f80130ed1a","Type":"ContainerStarted","Data":"8d444b6f534aaa5d606464fc4421ae6c534a5d9735699822211fce0cb683ae35"}
May 26 12:37:37.946253 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:37.946219    2563 generic.go:358] "Generic (PLEG): container finished" podID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerID="4c4dac77941f6cfd07497f14dad2e04228762cf08c6b53badb0216faee60772a" exitCode=0
May 26 12:37:37.946590 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:37.946283    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" event={"ID":"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666","Type":"ContainerDied","Data":"4c4dac77941f6cfd07497f14dad2e04228762cf08c6b53badb0216faee60772a"}
May 26 12:37:37.973477 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:37.973456    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:37:38.070242 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.070164    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-cabundle-cert\") pod \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") "
May 26 12:37:38.070242 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.070213    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kserve-provision-location\") pod \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") "
May 26 12:37:38.070445 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.070241    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") "
May 26 12:37:38.070445 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.070295    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-proxy-tls\") pod \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") "
May 26 12:37:38.070445 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.070351    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmkxx\" (UniqueName: \"kubernetes.io/projected/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kube-api-access-cmkxx\") pod \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\" (UID: \"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666\") "
May 26 12:37:38.070636 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.070607    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" (UID: "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:37:38.070701 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.070620    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" (UID: "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:37:38.070701 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.070670    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" (UID: "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:37:38.072607 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.072573    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" (UID: "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:37:38.072685 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.072593    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kube-api-access-cmkxx" (OuterVolumeSpecName: "kube-api-access-cmkxx") pod "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" (UID: "d5a7e7c5-a826-4bdf-83f1-2a0c0a835666"). InnerVolumeSpecName "kube-api-access-cmkxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:37:38.170812 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.170752    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkxx\" (UniqueName: \"kubernetes.io/projected/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kube-api-access-cmkxx\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:38.170812 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.170783    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-cabundle-cert\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:38.170812 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.170820    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:38.171031 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.170834    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:38.171031 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.170850    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:38.950455 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.950420    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25" event={"ID":"d5a7e7c5-a826-4bdf-83f1-2a0c0a835666","Type":"ContainerDied","Data":"d0f152577f8826465992354ed8abc177125f9753f093ede28c25d70c13997db1"}
May 26 12:37:38.950455 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.950463    2563 scope.go:117] "RemoveContainer" containerID="9507536ed118442060063ba82f5e699972c940fb4168f93502f1f464f279791e"
May 26 12:37:38.950972 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.950472    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"
May 26 12:37:38.958342 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.958309    2563 scope.go:117] "RemoveContainer" containerID="4c4dac77941f6cfd07497f14dad2e04228762cf08c6b53badb0216faee60772a"
May 26 12:37:38.965471 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.965455    2563 scope.go:117] "RemoveContainer" containerID="8250b343817629da8b7dcb4891a00f38eb1f0c68bca14a7d0fd4eeb3da77bf7e"
May 26 12:37:38.968609 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.968587    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"]
May 26 12:37:38.972416 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:38.972394    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-l6f25"]
May 26 12:37:39.953994 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:39.953967    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t_fbabfa19-0d61-4afb-bd4e-79f80130ed1a/storage-initializer/0.log"
May 26 12:37:39.954388 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:39.954003    2563 generic.go:358] "Generic (PLEG): container finished" podID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerID="2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007" exitCode=1
May 26 12:37:39.954388 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:39.954059    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t" event={"ID":"fbabfa19-0d61-4afb-bd4e-79f80130ed1a","Type":"ContainerDied","Data":"2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007"}
May 26 12:37:40.242063 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:40.242025    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" path="/var/lib/kubelet/pods/d5a7e7c5-a826-4bdf-83f1-2a0c0a835666/volumes"
May 26 12:37:40.961544 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:40.961513    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t_fbabfa19-0d61-4afb-bd4e-79f80130ed1a/storage-initializer/0.log"
May 26 12:37:40.962038 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:40.961615    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t" event={"ID":"fbabfa19-0d61-4afb-bd4e-79f80130ed1a","Type":"ContainerStarted","Data":"064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222"}
May 26 12:37:44.756433 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:44.756396    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"]
May 26 12:37:44.757044 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:44.756659    2563 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t" podUID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerName="storage-initializer" containerID="cri-o://064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222" gracePeriod=30
May 26 12:37:45.896060 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.896041    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t_fbabfa19-0d61-4afb-bd4e-79f80130ed1a/storage-initializer/1.log"
May 26 12:37:45.896400 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.896385    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t_fbabfa19-0d61-4afb-bd4e-79f80130ed1a/storage-initializer/0.log"
May 26 12:37:45.896456 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.896449    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:45.975044 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.975019    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t_fbabfa19-0d61-4afb-bd4e-79f80130ed1a/storage-initializer/1.log"
May 26 12:37:45.975361 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.975347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t_fbabfa19-0d61-4afb-bd4e-79f80130ed1a/storage-initializer/0.log"
May 26 12:37:45.975451 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.975379    2563 generic.go:358] "Generic (PLEG): container finished" podID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerID="064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222" exitCode=1
May 26 12:37:45.975451 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.975412    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t" event={"ID":"fbabfa19-0d61-4afb-bd4e-79f80130ed1a","Type":"ContainerDied","Data":"064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222"}
May 26 12:37:45.975451 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.975450    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t" event={"ID":"fbabfa19-0d61-4afb-bd4e-79f80130ed1a","Type":"ContainerDied","Data":"8d444b6f534aaa5d606464fc4421ae6c534a5d9735699822211fce0cb683ae35"}
May 26 12:37:45.975613 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.975467    2563 scope.go:117] "RemoveContainer" containerID="064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222"
May 26 12:37:45.975613 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.975481    2563 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"
May 26 12:37:45.982599 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.982580    2563 scope.go:117] "RemoveContainer" containerID="2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007"
May 26 12:37:45.993614 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.993592    2563 scope.go:117] "RemoveContainer" containerID="064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222"
May 26 12:37:45.993983 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:37:45.993955    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222\": container with ID starting with 064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222 not found: ID does not exist" containerID="064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222"
May 26 12:37:45.994065 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.993998    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222"} err="failed to get container status \"064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222\": rpc error: code = NotFound desc = could not find container \"064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222\": container with ID starting with 064af312dba01205a082b99453caf28011acac6bc0c98b51634d7d93db0c7222 not found: ID does not exist"
May 26 12:37:45.994065 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.994027    2563 scope.go:117] "RemoveContainer" containerID="2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007"
May 26 12:37:45.994333 ip-10-0-138-207 kubenswrapper[2563]: E0526 12:37:45.994315    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007\": container with ID starting with 2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007 not found: ID does not exist" containerID="2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007"
May 26 12:37:45.994388 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:45.994340    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007"} err="failed to get container status \"2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007\": rpc error: code = NotFound desc = could not find container \"2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007\": container with ID starting with 2e64cedeb17469513f3b5fc82a4e5ed0acbdf05544f0069eae87dd7edde98007 not found: ID does not exist"
May 26 12:37:46.029622 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.029595    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kserve-provision-location\") pod \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") "
May 26 12:37:46.029743 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.029629    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg8h7\" (UniqueName: \"kubernetes.io/projected/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kube-api-access-gg8h7\") pod \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") "
May 26 12:37:46.029743 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.029655    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") "
May 26 12:37:46.029743 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.029680    2563 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls\") pod \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\" (UID: \"fbabfa19-0d61-4afb-bd4e-79f80130ed1a\") "
May 26 12:37:46.029918 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.029893    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fbabfa19-0d61-4afb-bd4e-79f80130ed1a" (UID: "fbabfa19-0d61-4afb-bd4e-79f80130ed1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:37:46.029997 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.029978    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "fbabfa19-0d61-4afb-bd4e-79f80130ed1a" (UID: "fbabfa19-0d61-4afb-bd4e-79f80130ed1a"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 12:37:46.031691 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.031670    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kube-api-access-gg8h7" (OuterVolumeSpecName: "kube-api-access-gg8h7") pod "fbabfa19-0d61-4afb-bd4e-79f80130ed1a" (UID: "fbabfa19-0d61-4afb-bd4e-79f80130ed1a"). InnerVolumeSpecName "kube-api-access-gg8h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:37:46.031823 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.031786    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbabfa19-0d61-4afb-bd4e-79f80130ed1a" (UID: "fbabfa19-0d61-4afb-bd4e-79f80130ed1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:37:46.130569 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.130531    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kserve-provision-location\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:46.130569 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.130562    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg8h7\" (UniqueName: \"kubernetes.io/projected/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-kube-api-access-gg8h7\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:46.130569 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.130573    2563 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:46.130826 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.130584    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbabfa19-0d61-4afb-bd4e-79f80130ed1a-proxy-tls\") on node \"ip-10-0-138-207.ec2.internal\" DevicePath \"\""
May 26 12:37:46.306586 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.306553    2563 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"]
May 26 12:37:46.312358 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:46.312332    2563 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-z7m4t"]
May 26 12:37:48.246013 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:37:48.243003    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" path="/var/lib/kubelet/pods/fbabfa19-0d61-4afb-bd4e-79f80130ed1a/volumes"
May 26 12:38:01.515672 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:01.515642    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:38:01.521659 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:01.521635    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:38:16.113415 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.113375    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-964gc/must-gather-bhqg5"]
May 26 12:38:16.114480 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.114457    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="storage-initializer"
May 26 12:38:16.114620 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.114609    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="storage-initializer"
May 26 12:38:16.114731 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.114721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerName="storage-initializer"
May 26 12:38:16.114828 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.114818    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerName="storage-initializer"
May 26 12:38:16.114928 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.114919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kube-rbac-proxy"
May 26 12:38:16.114997 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.114989    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kube-rbac-proxy"
May 26 12:38:16.115072 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.115063    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container"
May 26 12:38:16.115141 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.115133    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container"
May 26 12:38:16.115217 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.115207    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerName="storage-initializer"
May 26 12:38:16.115287 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.115279    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerName="storage-initializer"
May 26 12:38:16.115471 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.115460    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kserve-container"
May 26 12:38:16.115560 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.115552    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5a7e7c5-a826-4bdf-83f1-2a0c0a835666" containerName="kube-rbac-proxy"
May 26 12:38:16.115634 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.115624    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerName="storage-initializer"
May 26 12:38:16.115721 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.115712    2563 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbabfa19-0d61-4afb-bd4e-79f80130ed1a" containerName="storage-initializer"
May 26 12:38:16.119437 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.119413    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-964gc/must-gather-bhqg5"
May 26 12:38:16.123703 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.123673    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-964gc\"/\"openshift-service-ca.crt\""
May 26 12:38:16.124223 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.124206    2563 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-964gc\"/\"default-dockercfg-tzckk\""
May 26 12:38:16.124329 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.124259    2563 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-964gc\"/\"kube-root-ca.crt\""
May 26 12:38:16.127198 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.127176    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-964gc/must-gather-bhqg5"]
May 26 12:38:16.144205 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.144180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xrbz\" (UniqueName: \"kubernetes.io/projected/ae91abcf-1958-4915-bc7f-b20fce96f3b5-kube-api-access-5xrbz\") pod \"must-gather-bhqg5\" (UID: \"ae91abcf-1958-4915-bc7f-b20fce96f3b5\") " pod="openshift-must-gather-964gc/must-gather-bhqg5"
May 26 12:38:16.144301 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.144218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae91abcf-1958-4915-bc7f-b20fce96f3b5-must-gather-output\") pod \"must-gather-bhqg5\" (UID: \"ae91abcf-1958-4915-bc7f-b20fce96f3b5\") " pod="openshift-must-gather-964gc/must-gather-bhqg5"
May 26 12:38:16.245025 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.245002    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrbz\" (UniqueName: \"kubernetes.io/projected/ae91abcf-1958-4915-bc7f-b20fce96f3b5-kube-api-access-5xrbz\") pod \"must-gather-bhqg5\" (UID: \"ae91abcf-1958-4915-bc7f-b20fce96f3b5\") " pod="openshift-must-gather-964gc/must-gather-bhqg5"
May 26 12:38:16.245182 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.245038    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae91abcf-1958-4915-bc7f-b20fce96f3b5-must-gather-output\") pod \"must-gather-bhqg5\" (UID: \"ae91abcf-1958-4915-bc7f-b20fce96f3b5\") " pod="openshift-must-gather-964gc/must-gather-bhqg5"
May 26 12:38:16.245327 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.245309    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae91abcf-1958-4915-bc7f-b20fce96f3b5-must-gather-output\") pod \"must-gather-bhqg5\" (UID: \"ae91abcf-1958-4915-bc7f-b20fce96f3b5\") " pod="openshift-must-gather-964gc/must-gather-bhqg5"
May 26 12:38:16.253332 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.253315    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrbz\" (UniqueName: \"kubernetes.io/projected/ae91abcf-1958-4915-bc7f-b20fce96f3b5-kube-api-access-5xrbz\") pod \"must-gather-bhqg5\" (UID: \"ae91abcf-1958-4915-bc7f-b20fce96f3b5\") " pod="openshift-must-gather-964gc/must-gather-bhqg5"
May 26 12:38:16.429204 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.429122    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-964gc/must-gather-bhqg5"
May 26 12:38:16.550457 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:16.550424    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-964gc/must-gather-bhqg5"]
May 26 12:38:16.553624 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:38:16.553585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae91abcf_1958_4915_bc7f_b20fce96f3b5.slice/crio-19a5fad22ffe67ded2508820cff86e2e5a3eeb393e166037fd1e7136eff81d0f WatchSource:0}: Error finding container 19a5fad22ffe67ded2508820cff86e2e5a3eeb393e166037fd1e7136eff81d0f: Status 404 returned error can't find the container with id 19a5fad22ffe67ded2508820cff86e2e5a3eeb393e166037fd1e7136eff81d0f
May 26 12:38:17.050761 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:17.050726    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-964gc/must-gather-bhqg5" event={"ID":"ae91abcf-1958-4915-bc7f-b20fce96f3b5","Type":"ContainerStarted","Data":"19a5fad22ffe67ded2508820cff86e2e5a3eeb393e166037fd1e7136eff81d0f"}
May 26 12:38:18.056564 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:18.056531    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-964gc/must-gather-bhqg5" event={"ID":"ae91abcf-1958-4915-bc7f-b20fce96f3b5","Type":"ContainerStarted","Data":"1a2e2bb75758c9eb45c067d75c3f0b1580490894caa11f923b0b34883bcb3e4e"}
May 26 12:38:18.056564 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:18.056566    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-964gc/must-gather-bhqg5" event={"ID":"ae91abcf-1958-4915-bc7f-b20fce96f3b5","Type":"ContainerStarted","Data":"0985f6c1558cfd6fdb7bd883a9279ec3e9f57a951a9b4f845b26e0cb0b37bf4e"}
May 26 12:38:18.071355 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:18.071303    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-964gc/must-gather-bhqg5" podStartSLOduration=1.235394339 podStartE2EDuration="2.071283088s" podCreationTimestamp="2026-05-26 12:38:16 +0000 UTC" firstStartedPulling="2026-05-26 12:38:16.555362447 +0000 UTC m=+4258.918312094" lastFinishedPulling="2026-05-26 12:38:17.391251194 +0000 UTC m=+4259.754200843" observedRunningTime="2026-05-26 12:38:18.070141344 +0000 UTC m=+4260.433091013" watchObservedRunningTime="2026-05-26 12:38:18.071283088 +0000 UTC m=+4260.434232758"
May 26 12:38:18.850786 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:18.850761    2563 ???:1] "http: TLS handshake error from 10.0.133.99:37414: EOF"
May 26 12:38:18.863407 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:18.863375    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-99q8l_b8f27af5-0b34-4fe2-ad62-afca8eb49a7c/global-pull-secret-syncer/0.log"
May 26 12:38:19.072207 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:19.072076    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bh49d_bc9937cb-665a-4714-b717-b577cedcd50f/konnectivity-agent/0.log"
May 26 12:38:19.176489 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:19.176411    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-207.ec2.internal_6decc4be96339c77a5112ef19602d9b6/haproxy/0.log"
May 26 12:38:23.174355 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:23.174326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdkbk_6479af9e-2fae-4bcf-bcad-bb978c70c652/node-exporter/0.log"
May 26 12:38:23.197426 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:23.197400    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdkbk_6479af9e-2fae-4bcf-bcad-bb978c70c652/kube-rbac-proxy/0.log"
May 26 12:38:23.221432 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:23.221410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdkbk_6479af9e-2fae-4bcf-bcad-bb978c70c652/init-textfile/0.log"
May 26 12:38:25.782918 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:25.782886    2563 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"]
May 26 12:38:25.785279 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:25.785256    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:25.793250 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:25.792532    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"]
May 26 12:38:25.927118 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:25.927079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftcn\" (UniqueName: \"kubernetes.io/projected/d6063bb4-22e6-4280-a6e7-70dce5bcc749-kube-api-access-gftcn\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:25.927404 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:25.927386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-sys\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:25.927550 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:25.927536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-lib-modules\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:25.927726 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:25.927714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-proc\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:25.927879 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:25.927862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-podres\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.029409 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.029369    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-proc\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.029770 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.029748    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-podres\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.030041 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.030012    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gftcn\" (UniqueName: \"kubernetes.io/projected/d6063bb4-22e6-4280-a6e7-70dce5bcc749-kube-api-access-gftcn\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.030151 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.030065    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-sys\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.030151 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.030093    2563 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-lib-modules\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.030151 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.029696    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-proc\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.030151 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.030015    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-podres\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.030306 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.030171    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-sys\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.030306 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.030263    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6063bb4-22e6-4280-a6e7-70dce5bcc749-lib-modules\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.038715 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.038651    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftcn\" (UniqueName: \"kubernetes.io/projected/d6063bb4-22e6-4280-a6e7-70dce5bcc749-kube-api-access-gftcn\") pod \"perf-node-gather-daemonset-pz86z\" (UID: \"d6063bb4-22e6-4280-a6e7-70dce5bcc749\") " pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.098151 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.098112    2563 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:26.252279 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.252257    2563 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"]
May 26 12:38:26.254511 ip-10-0-138-207 kubenswrapper[2563]: W0526 12:38:26.254477    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd6063bb4_22e6_4280_a6e7_70dce5bcc749.slice/crio-c19a4324cf982edc3e3752745e596190af9dcc7c94a04c6e5328693574bd8b04 WatchSource:0}: Error finding container c19a4324cf982edc3e3752745e596190af9dcc7c94a04c6e5328693574bd8b04: Status 404 returned error can't find the container with id c19a4324cf982edc3e3752745e596190af9dcc7c94a04c6e5328693574bd8b04
May 26 12:38:26.793201 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.793177    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-85gbr_c0d91a50-338c-450a-a121-2b08b9204793/dns/0.log"
May 26 12:38:26.811269 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.811239    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-85gbr_c0d91a50-338c-450a-a121-2b08b9204793/kube-rbac-proxy/0.log"
May 26 12:38:26.900329 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:26.900297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6t2bn_8eb48a6b-da7a-4dc2-963e-532b398e5974/dns-node-resolver/0.log"
May 26 12:38:27.089221 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:27.089129    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z" event={"ID":"d6063bb4-22e6-4280-a6e7-70dce5bcc749","Type":"ContainerStarted","Data":"7138dcd336e7ca3edae8e9c5c53f9c146d388f9ab635b433de3a69464d11e392"}
May 26 12:38:27.089221 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:27.089173    2563 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z" event={"ID":"d6063bb4-22e6-4280-a6e7-70dce5bcc749","Type":"ContainerStarted","Data":"c19a4324cf982edc3e3752745e596190af9dcc7c94a04c6e5328693574bd8b04"}
May 26 12:38:27.089464 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:27.089269    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:27.110849 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:27.107303    2563 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z" podStartSLOduration=2.107281038 podStartE2EDuration="2.107281038s" podCreationTimestamp="2026-05-26 12:38:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:38:27.104848952 +0000 UTC m=+4269.467798619" watchObservedRunningTime="2026-05-26 12:38:27.107281038 +0000 UTC m=+4269.470230708"
May 26 12:38:27.404391 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:27.404299    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4bwpv_132f1313-f019-435a-98ca-62608bdbbecd/node-ca/0.log"
May 26 12:38:28.418563 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:28.418519    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6d6tg_263e3474-ec60-4909-8953-24916d1eb74a/serve-healthcheck-canary/0.log"
May 26 12:38:28.883809 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:28.883758    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh992_92175d4e-f24c-4ac0-ab9c-ab1290c39388/kube-rbac-proxy/0.log"
May 26 12:38:28.902547 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:28.902516    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh992_92175d4e-f24c-4ac0-ab9c-ab1290c39388/exporter/0.log"
May 26 12:38:28.924199 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:28.924177    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh992_92175d4e-f24c-4ac0-ab9c-ab1290c39388/extractor/0.log"
May 26 12:38:31.044450 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:31.044411    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7cd9b78f5d-wnslt_e9f537db-2d0b-4a94-a27d-ec5c0baa569a/manager/0.log"
May 26 12:38:31.085574 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:31.085545    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-2khfb_7dd8a2ed-f55f-409c-956b-90085f39a31c/server/0.log"
May 26 12:38:31.576458 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:31.576425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-279st_1d3531b3-ada3-4f07-ae52-3c63415bad4e/seaweedfs/0.log"
May 26 12:38:33.105012 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:33.104985    2563 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-964gc/perf-node-gather-daemonset-pz86z"
May 26 12:38:37.037736 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.037645    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k9bkq_2a46dace-93be-4609-85f5-1682943deff4/kube-multus-additional-cni-plugins/0.log"
May 26 12:38:37.068877 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.068848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k9bkq_2a46dace-93be-4609-85f5-1682943deff4/egress-router-binary-copy/0.log"
May 26 12:38:37.093245 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.093213    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k9bkq_2a46dace-93be-4609-85f5-1682943deff4/cni-plugins/0.log"
May 26 12:38:37.119425 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.119399    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k9bkq_2a46dace-93be-4609-85f5-1682943deff4/bond-cni-plugin/0.log"
May 26 12:38:37.139737 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.139715    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k9bkq_2a46dace-93be-4609-85f5-1682943deff4/routeoverride-cni/0.log"
May 26 12:38:37.161158 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.161139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k9bkq_2a46dace-93be-4609-85f5-1682943deff4/whereabouts-cni-bincopy/0.log"
May 26 12:38:37.216365 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.216333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k9bkq_2a46dace-93be-4609-85f5-1682943deff4/whereabouts-cni/0.log"
May 26 12:38:37.414268 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.414238    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fbpkr_c260307a-5e7d-484e-ad49-0844501c96e1/kube-multus/0.log"
May 26 12:38:37.520144 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.520087    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sp6l5_7712945a-7f82-4e15-8b15-38ac03441d91/network-metrics-daemon/0.log"
May 26 12:38:37.534174 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:37.534148    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sp6l5_7712945a-7f82-4e15-8b15-38ac03441d91/kube-rbac-proxy/0.log"
May 26 12:38:38.947780 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:38.947745    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-controller/0.log"
May 26 12:38:38.965869 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:38.965844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/0.log"
May 26 12:38:39.005106 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:39.005074    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovn-acl-logging/1.log"
May 26 12:38:39.048098 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:39.048070    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/kube-rbac-proxy-node/0.log"
May 26 12:38:39.072879 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:39.072851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/kube-rbac-proxy-ovn-metrics/0.log"
May 26 12:38:39.091045 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:39.091023    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/northd/0.log"
May 26 12:38:39.111588 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:39.111569    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/nbdb/0.log"
May 26 12:38:39.130244 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:39.130219    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/sbdb/0.log"
May 26 12:38:39.327366 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:39.327332    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w89sb_8bab4db1-10bc-4fbf-945e-e0e88fca597b/ovnkube-controller/0.log"
May 26 12:38:40.292099 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:40.292070    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5tc55_f4d81d1d-211d-4896-b893-c8e472dbece8/network-check-target-container/0.log"
May 26 12:38:41.186171 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:41.186145    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-lx59v_3dcca82f-ff3d-4fe9-adeb-ca3b84741b77/iptables-alerter/0.log"
May 26 12:38:41.772566 ip-10-0-138-207 kubenswrapper[2563]: I0526 12:38:41.772538    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-fl6mp_cc7361ff-06fe-4836-b2f3-40ee04615ebb/tuned/0.log"