May 25 13:01:33.945847 ip-10-0-129-55 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 13:01:33.945862 ip-10-0-129-55 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 13:01:33.945871 ip-10-0-129-55 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 13:01:33.946227 ip-10-0-129-55 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 13:01:43.963110 ip-10-0-129-55 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 13:01:43.963128 ip-10-0-129-55 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 23eb9020c96348d3b563b1b6d38a65b7 --
May 25 13:04:09.854633 ip-10-0-129-55 systemd[1]: Starting Kubernetes Kubelet...
May 25 13:04:10.238000 ip-10-0-129-55 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:04:10.238000 ip-10-0-129-55 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 13:04:10.238000 ip-10-0-129-55 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:04:10.238000 ip-10-0-129-55 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 13:04:10.238000 ip-10-0-129-55 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:04:10.239265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.239048    2549 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 13:04:10.245297 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245265    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:10.245297 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245295    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245304    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245308    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245318    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245322    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245324    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245327    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245330    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245334    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245336    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245339    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245341    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245344    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245347    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245349    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245352    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245354    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245356    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245359    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245361    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:10.245383 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245365    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245367    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245370    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245372    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245375    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245377    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245381    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245383    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245386    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245391    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245395    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245398    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245401    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245404    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245407    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245409    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245412    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245415    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245417    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245419    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:10.245871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245422    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245424    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245427    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245429    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245431    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245434    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245436    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245438    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245441    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245443    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245445    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245448    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245450    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245452    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245456    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245459    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245462    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245464    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245467    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245470    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:10.246638 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245472    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245476    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245480    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245482    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245485    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245487    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245489    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245492    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245495    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245497    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245501    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245504    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245506    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245508    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245511    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245513    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245516    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245519    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245521    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:10.247220 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245524    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245526    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245529    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245531    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245534    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.245537    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.246838    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.246852    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.246856    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.246876    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.246879    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247005    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247008    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247011    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247014    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247017    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247020    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247022    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247025    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247028    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:10.247702 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247031    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247033    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247036    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247038    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247041    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247043    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247046    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247049    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247052    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247054    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247056    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247062    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247066    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247068    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247071    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247073    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247076    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247079    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247081    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247084    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:10.248303 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247087    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247090    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247093    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247100    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247107    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247110    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247113    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247116    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247119    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247122    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247125    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247128    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247131    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247134    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247136    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247139    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247141    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247144    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247147    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247149    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:10.248806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247151    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247154    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247157    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247160    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247163    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247166    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247168    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247171    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247173    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247176    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247178    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247181    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247184    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247186    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247189    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247191    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247193    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247196    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247198    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:10.249308 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247201    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247203    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247205    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247208    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247210    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247214    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247217    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247220    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247223    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247226    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247228    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247231    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.247233    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248446    2549 flags.go:64] FLAG: --address="0.0.0.0"
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248457    2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248465    2549 flags.go:64] FLAG: --anonymous-auth="true"
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248469    2549 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248474    2549 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248477    2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248484    2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248490    2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 13:04:10.249782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248493    2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248496    2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248500    2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248503    2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248507    2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248510    2549 flags.go:64] FLAG: --cgroup-root=""
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248512    2549 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248515    2549 flags.go:64] FLAG: --client-ca-file=""
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248518    2549 flags.go:64] FLAG: --cloud-config=""
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248521    2549 flags.go:64] FLAG: --cloud-provider="external"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.248524    2549 flags.go:64] FLAG: --cluster-dns="[]"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249159    2549 flags.go:64] FLAG: --cluster-domain=""
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249163    2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249167    2549 flags.go:64] FLAG: --config-dir=""
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249169    2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249174    2549 flags.go:64] FLAG: --container-log-max-files="5"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249178    2549 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249182    2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249186    2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249189    2549 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249192    2549 flags.go:64] FLAG: --contention-profiling="false"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249195    2549 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249198    2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249202    2549 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249204    2549 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 13:04:10.250324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249209    2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249212    2549 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249215    2549 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249219    2549 flags.go:64] FLAG: --enable-load-reader="false"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249222    2549 flags.go:64] FLAG: --enable-server="true"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249225    2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249230    2549 flags.go:64] FLAG: --event-burst="100"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249233    2549 flags.go:64] FLAG: --event-qps="50"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249236    2549 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249239    2549 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249242    2549 flags.go:64] FLAG: --eviction-hard=""
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249246    2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249250    2549 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249253    2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249256    2549 flags.go:64] FLAG: --eviction-soft=""
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249259    2549 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249262    2549 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249265    2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249287    2549 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249290    2549 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249293    2549 flags.go:64] FLAG: --fail-swap-on="true"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249296    2549 flags.go:64] FLAG: --feature-gates=""
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249301    2549 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249304    2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249309    2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 13:04:10.250908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249322    2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249326    2549 flags.go:64] FLAG: --healthz-port="10248"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249329    2549 flags.go:64] FLAG: --help="false"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249332    2549 flags.go:64] FLAG: --hostname-override="ip-10-0-129-55.ec2.internal"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249335    2549 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249339    2549 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249342    2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249345    2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249349    2549 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249352    2549 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249355    2549 flags.go:64] FLAG: --image-service-endpoint=""
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249358    2549 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249361    2549 flags.go:64] FLAG: --kube-api-burst="100"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249364    2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249367    2549 flags.go:64] FLAG: --kube-api-qps="50"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249370    2549 flags.go:64] FLAG: --kube-reserved=""
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249373    2549 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249376    2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249380    2549 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249383    2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249386    2549 flags.go:64] FLAG: --lock-file=""
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249389    2549 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249392    2549 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249395    2549 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 13:04:10.251526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249401    2549 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249404    2549 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249407    2549 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249409    2549 flags.go:64] FLAG: --logging-format="text"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249412    2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249416    2549 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249419    2549 flags.go:64] FLAG: --manifest-url=""
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249422    2549 flags.go:64] FLAG: --manifest-url-header=""
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249427    2549 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249431    2549 flags.go:64] FLAG: --max-open-files="1000000"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249439    2549 flags.go:64] FLAG: --max-pods="110"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249442    2549 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249446    2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249449    2549 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249452    2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249455    2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249458    2549 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249461    2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249470    2549 flags.go:64] FLAG: --node-status-max-images="50"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249474    2549 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249476    2549 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249480    2549 flags.go:64] FLAG: --pod-cidr=""
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249482    2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 13:04:10.252115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249488    2549 flags.go:64] FLAG: --pod-manifest-path=""
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249491    2549 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249494    2549 flags.go:64] FLAG: --pods-per-core="0"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249497    2549 flags.go:64] FLAG: --port="10250"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249500    2549 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249503    2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-037e60fa7fab93192"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249506    2549 flags.go:64] FLAG: --qos-reserved=""
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249509    2549 flags.go:64] FLAG: --read-only-port="10255"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249512    2549 flags.go:64] FLAG: --register-node="true"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249515    2549 flags.go:64] FLAG: --register-schedulable="true"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249518    2549 flags.go:64] FLAG: --register-with-taints=""
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249522    2549 flags.go:64] FLAG: --registry-burst="10"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249525    2549 flags.go:64] FLAG: --registry-qps="5"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249527    2549 flags.go:64] FLAG: --reserved-cpus=""
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249530    2549 flags.go:64] FLAG: --reserved-memory=""
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249534    2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249537    2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249544    2549 flags.go:64] FLAG: --rotate-certificates="false"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249547    2549 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249551    2549 flags.go:64] FLAG: --runonce="false"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249554    2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249558    2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249561    2549 flags.go:64] FLAG: --seccomp-default="false"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249564    2549 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249566    2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249570    2549 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 13:04:10.252685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249573    2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249576    2549 flags.go:64] FLAG: --storage-driver-password="root"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249579    2549 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249582    2549 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249584    2549 flags.go:64] FLAG: --storage-driver-user="root"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249587    2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249591    2549 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249593    2549 flags.go:64] FLAG: --system-cgroups=""
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249596    2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249602    2549 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249605    2549 flags.go:64] FLAG: --tls-cert-file=""
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249608    2549 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249612    2549 flags.go:64] FLAG: --tls-min-version=""
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249615    2549 flags.go:64] FLAG: --tls-private-key-file=""
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249618    2549 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249621    2549 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249623    2549 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249626    2549 flags.go:64] FLAG: --v="2"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249631    2549 flags.go:64] FLAG: --version="false"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249635    2549 flags.go:64] FLAG: --vmodule=""
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249640    2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.249643    2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249744    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249750    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:10.253367 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249753    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249756    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249760    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249763    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249765    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249768    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249771    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249773    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249776    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249779    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249782    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249784    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249787    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249790    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249793    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249795    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249798    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249800    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249803    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249806    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:10.253983 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249808    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249812    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249816    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249819    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249822    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249824    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249826    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249829    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249831    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249834    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249837    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249840    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249843    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249846    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249848    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249851    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249854    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249856    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249859    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:10.254509 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249861    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249864    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249866    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249869    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249871    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249874    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249876    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249878    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249881    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249883    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249886    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249888    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249890    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249893    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249895    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249898    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249900    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249902    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249905    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249907    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249910    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:10.254982 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249912    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249914    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249917    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249919    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249922    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249926    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249929    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249932    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249935    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249937    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249940    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249942    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249944    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249947    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249949    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249952    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249954    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249957    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249959    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249961    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:10.255551 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249964    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:10.256350 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249966    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:10.256350 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249968    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:10.256350 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.249971    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:10.256350 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.250614    2549 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:04:10.257402 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.257378    2549 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 13:04:10.257402 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.257402    2549 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257495    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257503    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257509    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257514    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257522    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257527    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257532    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257537    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257541    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257546    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:10.257548 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257551    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257555    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257560    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257566    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257571    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257576    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257580    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257584    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257589    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257592    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257597    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257601    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257604    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257608    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257612    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257616    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257621    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257625    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257630    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257634    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:10.258077 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257638    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257644    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257650    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257654    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257658    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257663    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257667    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257670    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257675    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257679    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257682    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257686    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257690    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257694    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257699    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257703    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257709    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257713    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257717    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:10.258884 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257721    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257726    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257730    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257733    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257738    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257743    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257748    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257752    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257756    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257761    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257765    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257770    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257773    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257777    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257781    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257786    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257790    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257794    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257798    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257803    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:10.259672 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257807    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257811    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257815    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257819    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257823    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257827    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257831    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257836    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257840    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257844    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257854    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257858    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257862    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257866    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257870    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257874    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:10.260496 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.257878    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.257886    2549 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258056    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258065    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258069    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258073    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258079    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258085    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258090    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258094    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258098    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258102    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258106    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258110    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258115    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:04:10.261031 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258119    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258124    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258128    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258132    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258136    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258140    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258144    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258148    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258153    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258157    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258161    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258165    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258170    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258174    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258178    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258183    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258187    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258191    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258195    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258199    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:04:10.261471 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258203    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258207    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258211    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258215    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258218    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258222    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258226    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258230    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258234    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258238    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258242    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258247    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258251    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258255    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258258    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258262    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258293    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258299    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258303    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258307    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:04:10.262086 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258312    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258316    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258320    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258324    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258328    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258332    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258337    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258341    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258345    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258349    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258353    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258357    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258361    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258365    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258369    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258373    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258377    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258381    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258384    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:04:10.262692 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258389    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258395    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258400    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258405    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258410    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258414    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258418    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258423    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258427    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258431    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258435    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258439    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258443    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:10.258447    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.258456    2549 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:04:10.263188 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.258670    2549 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 13:04:10.263580 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.261542    2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 13:04:10.263580 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.262360    2549 server.go:1019] "Starting client certificate rotation"
May 25 13:04:10.263580 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.262459    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:04:10.263580 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.262501    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:04:10.283185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.283159    2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:04:10.293606 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.286040    2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:04:10.305732 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.305707    2549 log.go:25] "Validated CRI v1 runtime API"
May 25 13:04:10.310892 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.310872    2549 log.go:25] "Validated CRI v1 image API"
May 25 13:04:10.312174 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.312159    2549 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 13:04:10.315107 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.315083    2549 fs.go:135] Filesystem UUIDs: map[27590eec-3fab-4828-bbe5-a7f121794702:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 de6d02b5-f607-4c91-9357-c45af0593db6:/dev/nvme0n1p3]
May 25 13:04:10.315187 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.315105    2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 13:04:10.318496 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.318479    2549 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:04:10.319767 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.319659    2549 manager.go:217] Machine: {Timestamp:2026-05-25 13:04:10.319153229 +0000 UTC m=+0.357910343 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099363 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ade84a91520102a910c1545d70b9f SystemUUID:ec2ade84-a915-2010-2a91-0c1545d70b9f BootID:23eb9020-c963-48d3-b563-b1b6d38a65b7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:b6:9f:4c:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:b6:9f:4c:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:39:32:86:39:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 13:04:10.320294 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.320264    2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 13:04:10.320428 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.320415    2549 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 13:04:10.323330 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.323303    2549 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 13:04:10.323471 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.323333    2549 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 13:04:10.323515 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.323481    2549 topology_manager.go:138] "Creating topology manager with none policy"
May 25 13:04:10.323515 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.323490    2549 container_manager_linux.go:306] "Creating device plugin manager"
May 25 13:04:10.323515 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.323498    2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:04:10.324038 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.324028    2549 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:04:10.326083 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.326071    2549 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:04:10.326194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.326185    2549 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 13:04:10.328037 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.328028    2549 kubelet.go:491] "Attempting to sync node with API server"
May 25 13:04:10.328075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.328048    2549 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 13:04:10.328075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.328059    2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 13:04:10.328075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.328069    2549 kubelet.go:397] "Adding apiserver pod source"
May 25 13:04:10.328184 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.328078    2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 13:04:10.329124 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.329112    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:04:10.329169 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.329132    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:04:10.333720 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.333704    2549 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 13:04:10.335610 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.335596    2549 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 13:04:10.336282 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336256    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 13:04:10.336323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336289    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 13:04:10.336323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336296    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 13:04:10.336323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336301    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 13:04:10.336323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336309    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 13:04:10.336323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336314    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 13:04:10.336323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336320    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 13:04:10.336323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336325    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 13:04:10.336500 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336332    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 13:04:10.336500 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336344    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 13:04:10.336500 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336356    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 13:04:10.336722 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.336714    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 13:04:10.337362 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.337353    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 13:04:10.337362 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.337362    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 13:04:10.342160 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.342145    2549 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 13:04:10.342227 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.342188    2549 server.go:1295] "Started kubelet"
May 25 13:04:10.342313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.342254    2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 13:04:10.342769 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.342714    2549 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 13:04:10.342872 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.342789    2549 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 13:04:10.343183 ip-10-0-129-55 systemd[1]: Started Kubernetes Kubelet.
May 25 13:04:10.346024 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.345998    2549 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 13:04:10.347405 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.347383    2549 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 13:04:10.347545 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.347415    2549 server.go:317] "Adding debug handlers to kubelet server"
May 25 13:04:10.347739 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.347712    2549 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 13:04:10.353308 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.353263    2549 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 13:04:10.354015 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.353287    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-55.ec2.internal.18b2d0ae6d58515b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-55.ec2.internal,UID:ip-10-0-129-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-55.ec2.internal,},FirstTimestamp:2026-05-25 13:04:10.342158683 +0000 UTC m=+0.380915798,LastTimestamp:2026-05-25 13:04:10.342158683 +0000 UTC m=+0.380915798,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-55.ec2.internal,}"
May 25 13:04:10.355581 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.355558    2549 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 13:04:10.356498 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.356483    2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 13:04:10.356582 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.356501    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 13:04:10.356582 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.356561    2549 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 13:04:10.356582 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.356570    2549 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 13:04:10.356721 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.356669    2549 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 13:04:10.356766 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.356749    2549 reconstruct.go:97] "Volume reconstruction finished"
May 25 13:04:10.356766 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.356761    2549 reconciler.go:26] "Reconciler: start to sync state"
May 25 13:04:10.357259 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.357240    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:10.357523 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.357507    2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 13:04:10.357584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.357527    2549 factory.go:55] Registering systemd factory
May 25 13:04:10.357584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.357537    2549 factory.go:223] Registration of the systemd container factory successfully
May 25 13:04:10.357773 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.357743    2549 factory.go:153] Registering CRI-O factory
May 25 13:04:10.357773 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.357765    2549 factory.go:223] Registration of the crio container factory successfully
May 25 13:04:10.357873 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.357792    2549 factory.go:103] Registering Raw factory
May 25 13:04:10.357873 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.357805    2549 manager.go:1196] Started watching for new ooms in manager
May 25 13:04:10.358244 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.358230    2549 manager.go:319] Starting recovery of all containers
May 25 13:04:10.360092 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.360071    2549 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 13:04:10.365679 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.365638    2549 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 13:04:10.366948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.366917    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5jv2"
May 25 13:04:10.372518 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.372502    2549 manager.go:324] Recovery completed
May 25 13:04:10.373437 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.373421    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5jv2"
May 25 13:04:10.376723 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.376709    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:10.379059 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.379040    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:10.379135 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.379078    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:10.379135 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.379092    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:10.379630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.379617    2549 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 13:04:10.379688 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.379632    2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 13:04:10.379688 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.379648    2549 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:04:10.381325 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.381232    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-55.ec2.internal.18b2d0ae6f8b602b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-55.ec2.internal,UID:ip-10-0-129-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-55.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-55.ec2.internal,},FirstTimestamp:2026-05-25 13:04:10.379059243 +0000 UTC m=+0.417816372,LastTimestamp:2026-05-25 13:04:10.379059243 +0000 UTC m=+0.417816372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-55.ec2.internal,}"
May 25 13:04:10.382452 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.382438    2549 policy_none.go:49] "None policy: Start"
May 25 13:04:10.382540 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.382492    2549 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 13:04:10.382540 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.382505    2549 state_mem.go:35] "Initializing new in-memory state store"
May 25 13:04:10.426074 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.426042    2549 manager.go:341] "Starting Device Plugin manager"
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.426123    2549 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.426138    2549 server.go:85] "Starting device plugin registration server"
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.426406    2549 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.426420    2549 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.426524    2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.426607    2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.426615    2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.427431    2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 13:04:10.435397 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.427472    2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:10.489865 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.489780    2549 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 13:04:10.491117 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.491098    2549 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 13:04:10.491178 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.491129    2549 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 13:04:10.491178 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.491156    2549 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 13:04:10.491178 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.491166    2549 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 13:04:10.491320 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.491263    2549 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 13:04:10.493241 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.493195    2549 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:10.527359 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.527315    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:10.528332 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.528311    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:10.528438 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.528341    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:10.528438 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.528353    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:10.528438 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.528376    2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-55.ec2.internal"
May 25 13:04:10.538951 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.538934    2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-55.ec2.internal"
May 25 13:04:10.539024 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.538958    2549 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-55.ec2.internal\": node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:10.557927 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.557905    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:10.591719 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.591690    2549 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal"]
May 25 13:04:10.591873 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.591767    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:10.592713 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.592698    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:10.592789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.592731    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:10.592789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.592748    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:10.593946 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.593931    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:10.594115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.594101    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.594164 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.594134    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:10.594706 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.594691    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:10.594774 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.594718    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:10.594774 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.594730    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:10.594774 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.594741    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:10.594774 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.594760    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:10.594774 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.594769    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:10.595898 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.595885    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.595943 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.595910    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:04:10.596713 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.596698    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:04:10.596786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.596719    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:04:10.596786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.596728    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeHasSufficientPID"
May 25 13:04:10.625082 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.625052    2549 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-55.ec2.internal\" not found" node="ip-10-0-129-55.ec2.internal"
May 25 13:04:10.628597 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.628577    2549 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-55.ec2.internal\" not found" node="ip-10-0-129-55.ec2.internal"
May 25 13:04:10.658499 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.658467    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:10.658634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.658528    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/085c1fe23cb3761f6985c2e4863c5818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal\" (UID: \"085c1fe23cb3761f6985c2e4863c5818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.658634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.658560    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c1fe23cb3761f6985c2e4863c5818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal\" (UID: \"085c1fe23cb3761f6985c2e4863c5818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.658634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.658579    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95dfbc508c808d021aadda17116338a3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-55.ec2.internal\" (UID: \"95dfbc508c808d021aadda17116338a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.758749 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.758662    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:10.758749 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.758746    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/085c1fe23cb3761f6985c2e4863c5818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal\" (UID: \"085c1fe23cb3761f6985c2e4863c5818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.758948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.758788    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c1fe23cb3761f6985c2e4863c5818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal\" (UID: \"085c1fe23cb3761f6985c2e4863c5818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.758948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.758829    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/085c1fe23cb3761f6985c2e4863c5818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal\" (UID: \"085c1fe23cb3761f6985c2e4863c5818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.758948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.758850    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085c1fe23cb3761f6985c2e4863c5818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal\" (UID: \"085c1fe23cb3761f6985c2e4863c5818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.758948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.758873    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95dfbc508c808d021aadda17116338a3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-55.ec2.internal\" (UID: \"95dfbc508c808d021aadda17116338a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.758948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.758897    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95dfbc508c808d021aadda17116338a3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-55.ec2.internal\" (UID: \"95dfbc508c808d021aadda17116338a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.859341 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.859300    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:10.928525 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.928498    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.932372 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:10.932314    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal"
May 25 13:04:10.959953 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:10.959913    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:11.060422 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:11.060319    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:11.160738 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:11.160699    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:11.261205 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:11.261165    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:11.262291 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.262257    2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 13:04:11.262447 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.262431    2549 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:04:11.357154 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.357068    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 13:04:11.361401 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:11.361381    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:11.368765 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.368741    2549 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:04:11.375531 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.375502    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 12:59:10 +0000 UTC" deadline="2028-01-23 13:40:16.168190732 +0000 UTC"
May 25 13:04:11.375531 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.375529    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14592h36m4.792664576s"
May 25 13:04:11.385423 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.385401    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sp2st"
May 25 13:04:11.393728 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.393706    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sp2st"
May 25 13:04:11.427752 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.427726    2549 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:11.461999 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:11.461963    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:11.562531 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:11.562499    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-55.ec2.internal\" not found"
May 25 13:04:11.610642 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:11.610564    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dfbc508c808d021aadda17116338a3.slice/crio-7911beb1484b17733466cdb3de0dda2e9081a8d7f33a00fa8b2ccefa6625a39c WatchSource:0}: Error finding container 7911beb1484b17733466cdb3de0dda2e9081a8d7f33a00fa8b2ccefa6625a39c: Status 404 returned error can't find the container with id 7911beb1484b17733466cdb3de0dda2e9081a8d7f33a00fa8b2ccefa6625a39c
May 25 13:04:11.610866 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:11.610839    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085c1fe23cb3761f6985c2e4863c5818.slice/crio-d9a341d69f973daf7754a3854988694e01c28eb61e26dc51170d0dc2b22dec84 WatchSource:0}: Error finding container d9a341d69f973daf7754a3854988694e01c28eb61e26dc51170d0dc2b22dec84: Status 404 returned error can't find the container with id d9a341d69f973daf7754a3854988694e01c28eb61e26dc51170d0dc2b22dec84
May 25 13:04:11.614155 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.614142    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:04:11.660871 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.659615    2549 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:11.757793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.757756    2549 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal"
May 25 13:04:11.769860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.769824    2549 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:11.769997 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.769896    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:04:11.771257 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.771242    2549 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal"
May 25 13:04:11.779281 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:11.779254    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:04:12.127660 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.127627    2549 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:04:12.329099 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.329065    2549 apiserver.go:52] "Watching apiserver"
May 25 13:04:12.334081 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.334058    2549 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 13:04:12.337400 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.337368    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-qx45s","kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal","openshift-image-registry/node-ca-pqbcw","openshift-multus/multus-additional-cni-plugins-lw2vp","openshift-multus/network-metrics-daemon-kmndm","openshift-network-diagnostics/network-check-target-qnwqv","openshift-network-operator/iptables-alerter-2hc5v","kube-system/konnectivity-agent-8rwnv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r","openshift-cluster-node-tuning-operator/tuned-xs46z","openshift-dns/node-resolver-6hzsz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal","openshift-multus/multus-2mzsn"]
May 25 13:04:12.340001 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.339977    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.342103 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.342075    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.342370 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.342220    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xpm2v\""
May 25 13:04:12.342495 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.342090    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 13:04:12.342738 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.342722    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 13:04:12.343001 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.342972    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 13:04:12.344001 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.343976    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 13:04:12.344403 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.344380    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.344563 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.344539    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pthvb\""
May 25 13:04:12.344763 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.344751    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 13:04:12.345146 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.345130    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 13:04:12.346379 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.346361    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 13:04:12.346480 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.346390    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7z42x\""
May 25 13:04:12.346537 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.346522    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:12.346919 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.346594    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:12.346919 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.346685    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 13:04:12.346919 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.346706    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 13:04:12.346919 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.346721    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 13:04:12.346919 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.346828    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 13:04:12.348931 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.348912    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:12.349012 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.348983    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:12.352172 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.352153    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.353963 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.353945    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 13:04:12.354049 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.353986    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 13:04:12.354106 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.354053    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h8kwb\""
May 25 13:04:12.354542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.354315    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 13:04:12.354542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.354374    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 13:04:12.354542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.354393    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 13:04:12.354542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.354378    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 13:04:12.354790 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.354622    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:12.358232 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.358210    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 13:04:12.358582 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.358564    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 13:04:12.358672 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.358614    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5jdmq\""
May 25 13:04:12.359956 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.359935    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.360260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.359940    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.362225 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.362205    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 13:04:12.362458 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.362440    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8tf4t\""
May 25 13:04:12.362789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.362761    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 13:04:12.362879 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.362805    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.362879 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.362815    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 13:04:12.363026 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.363009    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 13:04:12.363196 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.363178    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-trdbl\""
May 25 13:04:12.364634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.364614    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 13:04:12.364726 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.364690    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 13:04:12.364841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.364825    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 13:04:12.364886 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.364858    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xrghl\""
May 25 13:04:12.365525 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365228    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.365525 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365483    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-cnibin\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.365525 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365521    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.365699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365545    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-systemd\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.365699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365584    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-node-log\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.365699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365629    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-cni-bin\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.365699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365653    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-cni-netd\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.365699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365677    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khs88\" (UniqueName: \"kubernetes.io/projected/218a9570-215b-4ca9-8d61-f9f39f8295e5-kube-api-access-khs88\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.365863 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365718    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-kubelet\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.365863 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365743    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.365863 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365768    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.365863 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365808    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovnkube-config\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.365863 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365832    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovnkube-script-lib\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.366054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365871    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:12.366054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365897    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.366054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365920    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.366054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365941    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-env-overrides\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.366054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365964    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b628t\" (UniqueName: \"kubernetes.io/projected/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-kube-api-access-b628t\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.366054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.365997    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b9ff95a-a2af-4375-beea-7e288a65ab11-host\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.366054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366021    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-system-cni-dir\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.366054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366043    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-slash\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366066    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feaa61e8-ce75-4474-8a0c-bc149dcb8074-konnectivity-ca\") pod \"konnectivity-agent-8rwnv\" (UID: \"feaa61e8-ce75-4474-8a0c-bc149dcb8074\") " pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366090    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366113    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-registration-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366153    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-socket-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366186    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-run-netns\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366213    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-log-socket\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366236    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovn-node-metrics-cert\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366260    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ck4\" (UniqueName: \"kubernetes.io/projected/22897fc3-4085-4592-92b7-ba00a4a8dfaa-kube-api-access-67ck4\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366303    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-device-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366333    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-sys-fs\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366369    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-ovn\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366420    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93dafc65-1e53-4bf2-a21c-c8da486adaea-iptables-alerter-script\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.366460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366462    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-os-release\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366506    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366529    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366555    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-systemd-units\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366611    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b9ff95a-a2af-4375-beea-7e288a65ab11-serviceca\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366637    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwfs\" (UniqueName: \"kubernetes.io/projected/2b9ff95a-a2af-4375-beea-7e288a65ab11-kube-api-access-gwwfs\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366664    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366689    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmwk4\" (UniqueName: \"kubernetes.io/projected/cf5b6105-22f6-4f92-9cd2-5078d464a323-kube-api-access-tmwk4\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366714    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-var-lib-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366731    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-etc-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366745    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366758    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feaa61e8-ce75-4474-8a0c-bc149dcb8074-agent-certs\") pod \"konnectivity-agent-8rwnv\" (UID: \"feaa61e8-ce75-4474-8a0c-bc149dcb8074\") " pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366771    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93dafc65-1e53-4bf2-a21c-c8da486adaea-host-slash\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.367055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.366785    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzh2t\" (UniqueName: \"kubernetes.io/projected/93dafc65-1e53-4bf2-a21c-c8da486adaea-kube-api-access-hzh2t\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.367636 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.367241    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v8gxk\""
May 25 13:04:12.367636 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.367286    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 13:04:12.396595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.396563    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:59:11 +0000 UTC" deadline="2027-10-30 23:09:19.500740991 +0000 UTC"
May 25 13:04:12.396595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.396593    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12562h5m7.104151203s"
May 25 13:04:12.461122 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.458821    2549 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 13:04:12.467370 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467326    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-cnibin\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.467370 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467373    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-node-log\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467397    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-cni-bin\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467419    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-cni-netd\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467446    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-var-lib-kubelet\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467452    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-cnibin\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467469    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467505    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467526    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467547    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-node-log\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467559    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-cni-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467580    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-cni-bin\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467584    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysctl-d\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467609    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a934d1e-67ac-490f-b795-9203848f808b-tmp\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.467621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467616    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-cni-netd\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467632    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c6d0a9e-64c4-44de-8429-196cdbb51c51-cni-binary-copy\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467658    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-netns\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467684    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-slash\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467710    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-registration-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467739    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-modprobe-d\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467764    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467790    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysconfig\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467818    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-run\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467845    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-run-netns\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467867    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-device-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467889    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-sys-fs\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467914    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-cnibin\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467948    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-hostroot\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467957    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-registration-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.467974    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-ovn\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468010    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-daemon-config\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.468364 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468059    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-slash\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468073    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-sys-fs\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468093    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-ovn\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468132    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1363056f-5cf2-4736-8032-7cbc4cd602e1-tmp-dir\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468144    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-device-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468159    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwfs\" (UniqueName: \"kubernetes.io/projected/2b9ff95a-a2af-4375-beea-7e288a65ab11-kube-api-access-gwwfs\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468187    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-etc-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468204    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-run-netns\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468215    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b628t\" (UniqueName: \"kubernetes.io/projected/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-kube-api-access-b628t\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468288    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-cni-bin\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468322    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-cni-multus\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468340    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-etc-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468351    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-etc-kubernetes\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468390    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-systemd\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468428    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-lib-modules\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468460    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hzh2t\" (UniqueName: \"kubernetes.io/projected/93dafc65-1e53-4bf2-a21c-c8da486adaea-kube-api-access-hzh2t\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468489    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.469056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468545    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-systemd\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468599    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468654    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-run-systemd\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468678    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-sys\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468727    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468744    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-host\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468783    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a934d1e-67ac-490f-b795-9203848f808b-etc-tuned\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468814    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhmc\" (UniqueName: \"kubernetes.io/projected/5a934d1e-67ac-490f-b795-9203848f808b-kube-api-access-hfhmc\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468844    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-khs88\" (UniqueName: \"kubernetes.io/projected/218a9570-215b-4ca9-8d61-f9f39f8295e5-kube-api-access-khs88\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468867    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-kubelet\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468948    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovnkube-config\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.468975    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovnkube-script-lib\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469000    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469048    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-conf-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469056    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-kubelet\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469075    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469126    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.469845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469185    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-env-overrides\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469223    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469325    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469356    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b9ff95a-a2af-4375-beea-7e288a65ab11-host\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469453    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-system-cni-dir\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469461    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b9ff95a-a2af-4375-beea-7e288a65ab11-host\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469492    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feaa61e8-ce75-4474-8a0c-bc149dcb8074-konnectivity-ca\") pod \"konnectivity-agent-8rwnv\" (UID: \"feaa61e8-ce75-4474-8a0c-bc149dcb8074\") " pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469503    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovnkube-config\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469564    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-system-cni-dir\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469563    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469621    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469652    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-kubernetes\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469690    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-socket-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469718    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-multus-certs\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469743    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-env-overrides\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469743    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysctl-conf\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469782    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-socket-dir-parent\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.470626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469808    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-log-socket\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469808    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-socket-dir\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469814    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovnkube-script-lib\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469825    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovn-node-metrics-cert\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469843    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-67ck4\" (UniqueName: \"kubernetes.io/projected/22897fc3-4085-4592-92b7-ba00a4a8dfaa-kube-api-access-67ck4\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469865    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-os-release\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469881    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-kubelet\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469896    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dk82\" (UniqueName: \"kubernetes.io/projected/8c6d0a9e-64c4-44de-8429-196cdbb51c51-kube-api-access-5dk82\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469913    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1363056f-5cf2-4736-8032-7cbc4cd602e1-hosts-file\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469862    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-log-socket\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469945    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-system-cni-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469970    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93dafc65-1e53-4bf2-a21c-c8da486adaea-iptables-alerter-script\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469980    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feaa61e8-ce75-4474-8a0c-bc149dcb8074-konnectivity-ca\") pod \"konnectivity-agent-8rwnv\" (UID: \"feaa61e8-ce75-4474-8a0c-bc149dcb8074\") " pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.469985    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-os-release\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470036    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/218a9570-215b-4ca9-8d61-f9f39f8295e5-os-release\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470044    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470075    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:12.471340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470098    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-systemd-units\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470121    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b9ff95a-a2af-4375-beea-7e288a65ab11-serviceca\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470145    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470171    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tmwk4\" (UniqueName: \"kubernetes.io/projected/cf5b6105-22f6-4f92-9cd2-5078d464a323-kube-api-access-tmwk4\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470153    2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470194    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-var-lib-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470253    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.470195    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.470352    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:12.970333681 +0000 UTC m=+3.009090798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470509    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93dafc65-1e53-4bf2-a21c-c8da486adaea-iptables-alerter-script\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470509    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470542    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-systemd-units\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470591    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470616    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feaa61e8-ce75-4474-8a0c-bc149dcb8074-agent-certs\") pod \"konnectivity-agent-8rwnv\" (UID: \"feaa61e8-ce75-4474-8a0c-bc149dcb8074\") " pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470647    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfg5\" (UniqueName: \"kubernetes.io/projected/1363056f-5cf2-4736-8032-7cbc4cd602e1-kube-api-access-lpfg5\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470676    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93dafc65-1e53-4bf2-a21c-c8da486adaea-host-slash\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470717    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/218a9570-215b-4ca9-8d61-f9f39f8295e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.471860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470764    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93dafc65-1e53-4bf2-a21c-c8da486adaea-host-slash\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.472388 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470799    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22897fc3-4085-4592-92b7-ba00a4a8dfaa-var-lib-openvswitch\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.472388 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.470857    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b9ff95a-a2af-4375-beea-7e288a65ab11-serviceca\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.473688 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.473665    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22897fc3-4085-4592-92b7-ba00a4a8dfaa-ovn-node-metrics-cert\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.474843 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.474755    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feaa61e8-ce75-4474-8a0c-bc149dcb8074-agent-certs\") pod \"konnectivity-agent-8rwnv\" (UID: \"feaa61e8-ce75-4474-8a0c-bc149dcb8074\") " pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:12.475730 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.475395    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:12.475730 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.475418    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:12.475730 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.475434    2549 projected.go:194] Error preparing data for projected volume kube-api-access-g2bj6 for pod openshift-network-diagnostics/network-check-target-qnwqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:12.475730 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.475498    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6 podName:10569d7e-c62f-4724-b920-daf76ac7365f nodeName:}" failed. No retries permitted until 2026-05-25 13:04:12.975482253 +0000 UTC m=+3.014239372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g2bj6" (UniqueName: "kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6") pod "network-check-target-qnwqv" (UID: "10569d7e-c62f-4724-b920-daf76ac7365f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:12.476952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.476927    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwfs\" (UniqueName: \"kubernetes.io/projected/2b9ff95a-a2af-4375-beea-7e288a65ab11-kube-api-access-gwwfs\") pod \"node-ca-pqbcw\" (UID: \"2b9ff95a-a2af-4375-beea-7e288a65ab11\") " pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.477405 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.477377    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b628t\" (UniqueName: \"kubernetes.io/projected/d7e58a46-c553-4bf1-96dc-ad68ddac1b4a-kube-api-access-b628t\") pod \"aws-ebs-csi-driver-node-m8c6r\" (UID: \"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.478353 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.478093    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khs88\" (UniqueName: \"kubernetes.io/projected/218a9570-215b-4ca9-8d61-f9f39f8295e5-kube-api-access-khs88\") pod \"multus-additional-cni-plugins-lw2vp\" (UID: \"218a9570-215b-4ca9-8d61-f9f39f8295e5\") " pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.478854 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.478828    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzh2t\" (UniqueName: \"kubernetes.io/projected/93dafc65-1e53-4bf2-a21c-c8da486adaea-kube-api-access-hzh2t\") pod \"iptables-alerter-2hc5v\" (UID: \"93dafc65-1e53-4bf2-a21c-c8da486adaea\") " pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.479393 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.479341    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmwk4\" (UniqueName: \"kubernetes.io/projected/cf5b6105-22f6-4f92-9cd2-5078d464a323-kube-api-access-tmwk4\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:12.481359 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.481320    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ck4\" (UniqueName: \"kubernetes.io/projected/22897fc3-4085-4592-92b7-ba00a4a8dfaa-kube-api-access-67ck4\") pod \"ovnkube-node-qx45s\" (UID: \"22897fc3-4085-4592-92b7-ba00a4a8dfaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.496353 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.496296    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal" event={"ID":"95dfbc508c808d021aadda17116338a3","Type":"ContainerStarted","Data":"7911beb1484b17733466cdb3de0dda2e9081a8d7f33a00fa8b2ccefa6625a39c"}
May 25 13:04:12.497368 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.497343    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal" event={"ID":"085c1fe23cb3761f6985c2e4863c5818","Type":"ContainerStarted","Data":"d9a341d69f973daf7754a3854988694e01c28eb61e26dc51170d0dc2b22dec84"}
May 25 13:04:12.571014 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.570971    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-socket-dir-parent\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571014 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571020    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-os-release\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571055    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-kubelet\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571082    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5dk82\" (UniqueName: \"kubernetes.io/projected/8c6d0a9e-64c4-44de-8429-196cdbb51c51-kube-api-access-5dk82\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571103    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1363056f-5cf2-4736-8032-7cbc4cd602e1-hosts-file\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.571265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571118    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-system-cni-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571135    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-socket-dir-parent\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571165    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfg5\" (UniqueName: \"kubernetes.io/projected/1363056f-5cf2-4736-8032-7cbc4cd602e1-kube-api-access-lpfg5\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.571265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571195    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-var-lib-kubelet\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.571265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571233    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-cni-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571313    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-system-cni-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571327    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-os-release\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571344    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-var-lib-kubelet\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571354    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysctl-d\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571332    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-cni-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571401    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1363056f-5cf2-4736-8032-7cbc4cd602e1-hosts-file\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571401    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a934d1e-67ac-490f-b795-9203848f808b-tmp\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571408    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-kubelet\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571466    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c6d0a9e-64c4-44de-8429-196cdbb51c51-cni-binary-copy\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571481    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysctl-d\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571498    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-netns\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571532    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-modprobe-d\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571557    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-netns\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571560    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571597    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysconfig\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571603    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571641    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-run\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.571703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571651    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysconfig\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571679    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-cnibin\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571705    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-hostroot\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571681    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-run\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571732    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-daemon-config\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571757    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1363056f-5cf2-4736-8032-7cbc4cd602e1-tmp-dir\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571776    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-hostroot\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571642    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-modprobe-d\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571777    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-cnibin\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571784    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-cni-bin\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571827    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-cni-multus\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571854    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-etc-kubernetes\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571877    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-systemd\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571896    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-cni-bin\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571936    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-var-lib-cni-multus\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571976    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-etc-kubernetes\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571983    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-lib-modules\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572020    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-systemd\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.572564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.571901    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-lib-modules\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572065    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-sys\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572068    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1363056f-5cf2-4736-8032-7cbc4cd602e1-tmp-dir\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572088    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-host\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572125    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-host\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572125    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a934d1e-67ac-490f-b795-9203848f808b-etc-tuned\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572160    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhmc\" (UniqueName: \"kubernetes.io/projected/5a934d1e-67ac-490f-b795-9203848f808b-kube-api-access-hfhmc\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572190    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-conf-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572232    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-kubernetes\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572258    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-multus-certs\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572302    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysctl-conf\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572439    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-sysctl-conf\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572438    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-daemon-config\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572469    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-multus-conf-dir\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572481    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-etc-kubernetes\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572493    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c6d0a9e-64c4-44de-8429-196cdbb51c51-host-run-multus-certs\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572521    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a934d1e-67ac-490f-b795-9203848f808b-sys\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.573260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.572851    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c6d0a9e-64c4-44de-8429-196cdbb51c51-cni-binary-copy\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.574699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.574671    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a934d1e-67ac-490f-b795-9203848f808b-tmp\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.574806 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.574701    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a934d1e-67ac-490f-b795-9203848f808b-etc-tuned\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.579657 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.579626    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhmc\" (UniqueName: \"kubernetes.io/projected/5a934d1e-67ac-490f-b795-9203848f808b-kube-api-access-hfhmc\") pod \"tuned-xs46z\" (UID: \"5a934d1e-67ac-490f-b795-9203848f808b\") " pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.579806 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.579784    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dk82\" (UniqueName: \"kubernetes.io/projected/8c6d0a9e-64c4-44de-8429-196cdbb51c51-kube-api-access-5dk82\") pod \"multus-2mzsn\" (UID: \"8c6d0a9e-64c4-44de-8429-196cdbb51c51\") " pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.580420 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.580385    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfg5\" (UniqueName: \"kubernetes.io/projected/1363056f-5cf2-4736-8032-7cbc4cd602e1-kube-api-access-lpfg5\") pod \"node-resolver-6hzsz\" (UID: \"1363056f-5cf2-4736-8032-7cbc4cd602e1\") " pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.654409 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.654311    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2hc5v"
May 25 13:04:12.666300 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.666256    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pqbcw"
May 25 13:04:12.676537 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.676512    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lw2vp"
May 25 13:04:12.682309 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.682289    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:12.689919 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.689897    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:12.697610 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.697592    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xs46z"
May 25 13:04:12.706228 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.706191    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r"
May 25 13:04:12.714789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.714770    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6hzsz"
May 25 13:04:12.720396 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.720377    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mzsn"
May 25 13:04:12.976528 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.976431    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:12.976528 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:12.976502    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:12.976736 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.976594    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:12.976736 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.976630    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:12.976736 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.976647    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:12.976736 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.976661    2549 projected.go:194] Error preparing data for projected volume kube-api-access-g2bj6 for pod openshift-network-diagnostics/network-check-target-qnwqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:12.976736 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.976669    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:13.976648474 +0000 UTC m=+4.015405580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:12.976736 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:12.976717    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6 podName:10569d7e-c62f-4724-b920-daf76ac7365f nodeName:}" failed. No retries permitted until 2026-05-25 13:04:13.976704034 +0000 UTC m=+4.015461150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2bj6" (UniqueName: "kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6") pod "network-check-target-qnwqv" (UID: "10569d7e-c62f-4724-b920-daf76ac7365f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:13.276986 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.276908    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22897fc3_4085_4592_92b7_ba00a4a8dfaa.slice/crio-54ead090d89e2bd5e7d5db832485477df079bc8c87a9ad6d24a761c820a1821a WatchSource:0}: Error finding container 54ead090d89e2bd5e7d5db832485477df079bc8c87a9ad6d24a761c820a1821a: Status 404 returned error can't find the container with id 54ead090d89e2bd5e7d5db832485477df079bc8c87a9ad6d24a761c820a1821a
May 25 13:04:13.278708 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.278658    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218a9570_215b_4ca9_8d61_f9f39f8295e5.slice/crio-cdf1756972f0e82800a6bcb6c1af7f0fd8ac4c7f6ad6dd0262faa5b0121cfbac WatchSource:0}: Error finding container cdf1756972f0e82800a6bcb6c1af7f0fd8ac4c7f6ad6dd0262faa5b0121cfbac: Status 404 returned error can't find the container with id cdf1756972f0e82800a6bcb6c1af7f0fd8ac4c7f6ad6dd0262faa5b0121cfbac
May 25 13:04:13.280324 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.280300    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e58a46_c553_4bf1_96dc_ad68ddac1b4a.slice/crio-1f1b82637913197f70c3e11cd746020133b0b9f0041b465c2bf13d154e2496a2 WatchSource:0}: Error finding container 1f1b82637913197f70c3e11cd746020133b0b9f0041b465c2bf13d154e2496a2: Status 404 returned error can't find the container with id 1f1b82637913197f70c3e11cd746020133b0b9f0041b465c2bf13d154e2496a2
May 25 13:04:13.282052 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.282031    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6d0a9e_64c4_44de_8429_196cdbb51c51.slice/crio-ae5c7a2e608cd6292cec0251d84f2d9a419cc9102f4e3c81a8695cdf28684426 WatchSource:0}: Error finding container ae5c7a2e608cd6292cec0251d84f2d9a419cc9102f4e3c81a8695cdf28684426: Status 404 returned error can't find the container with id ae5c7a2e608cd6292cec0251d84f2d9a419cc9102f4e3c81a8695cdf28684426
May 25 13:04:13.283390 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.283368    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dafc65_1e53_4bf2_a21c_c8da486adaea.slice/crio-a3c275f4ba8b66c690f3c761f5533cb07dd39b5dcd2d1af1bb70efaa839e626e WatchSource:0}: Error finding container a3c275f4ba8b66c690f3c761f5533cb07dd39b5dcd2d1af1bb70efaa839e626e: Status 404 returned error can't find the container with id a3c275f4ba8b66c690f3c761f5533cb07dd39b5dcd2d1af1bb70efaa839e626e
May 25 13:04:13.311755 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.311706    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaa61e8_ce75_4474_8a0c_bc149dcb8074.slice/crio-053a072bbb120985f50c6c5afaa883673851c6b23e9b65425d919462640da39f WatchSource:0}: Error finding container 053a072bbb120985f50c6c5afaa883673851c6b23e9b65425d919462640da39f: Status 404 returned error can't find the container with id 053a072bbb120985f50c6c5afaa883673851c6b23e9b65425d919462640da39f
May 25 13:04:13.314604 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.314575    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1363056f_5cf2_4736_8032_7cbc4cd602e1.slice/crio-bb5e5b46ae4a5089075335f4b5b39306c02ea6963b581f71d812b5b5750319d7 WatchSource:0}: Error finding container bb5e5b46ae4a5089075335f4b5b39306c02ea6963b581f71d812b5b5750319d7: Status 404 returned error can't find the container with id bb5e5b46ae4a5089075335f4b5b39306c02ea6963b581f71d812b5b5750319d7
May 25 13:04:13.314954 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.314928    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9ff95a_a2af_4375_beea_7e288a65ab11.slice/crio-c0ae59eb6f84b1d279ed1c08d71e26b93c40f9e7fa120d904d80367a6d86eb04 WatchSource:0}: Error finding container c0ae59eb6f84b1d279ed1c08d71e26b93c40f9e7fa120d904d80367a6d86eb04: Status 404 returned error can't find the container with id c0ae59eb6f84b1d279ed1c08d71e26b93c40f9e7fa120d904d80367a6d86eb04
May 25 13:04:13.315923 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:13.315905    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a934d1e_67ac_490f_b795_9203848f808b.slice/crio-12620df851185bc029ca1460e403f68f2d753350afa10fe2c7a9008c8d06def5 WatchSource:0}: Error finding container 12620df851185bc029ca1460e403f68f2d753350afa10fe2c7a9008c8d06def5: Status 404 returned error can't find the container with id 12620df851185bc029ca1460e403f68f2d753350afa10fe2c7a9008c8d06def5
May 25 13:04:13.396988 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.396944    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:59:11 +0000 UTC" deadline="2027-12-01 23:22:15.532499781 +0000 UTC"
May 25 13:04:13.396988 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.396984    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13330h18m2.135520643s"
May 25 13:04:13.492297 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.492253    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:13.492460 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:13.492400    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:13.500159 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.500122    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2hc5v" event={"ID":"93dafc65-1e53-4bf2-a21c-c8da486adaea","Type":"ContainerStarted","Data":"a3c275f4ba8b66c690f3c761f5533cb07dd39b5dcd2d1af1bb70efaa839e626e"}
May 25 13:04:13.501309 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.501286    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mzsn" event={"ID":"8c6d0a9e-64c4-44de-8429-196cdbb51c51","Type":"ContainerStarted","Data":"ae5c7a2e608cd6292cec0251d84f2d9a419cc9102f4e3c81a8695cdf28684426"}
May 25 13:04:13.502288 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.502250    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r" event={"ID":"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a","Type":"ContainerStarted","Data":"1f1b82637913197f70c3e11cd746020133b0b9f0041b465c2bf13d154e2496a2"}
May 25 13:04:13.503196 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.503178    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" event={"ID":"218a9570-215b-4ca9-8d61-f9f39f8295e5","Type":"ContainerStarted","Data":"cdf1756972f0e82800a6bcb6c1af7f0fd8ac4c7f6ad6dd0262faa5b0121cfbac"}
May 25 13:04:13.505115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.504655    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal" event={"ID":"95dfbc508c808d021aadda17116338a3","Type":"ContainerStarted","Data":"f1cf94041204312684a9193835c17dbc741f08519dd69a37893b163f0be42f6b"}
May 25 13:04:13.505863 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.505833    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xs46z" event={"ID":"5a934d1e-67ac-490f-b795-9203848f808b","Type":"ContainerStarted","Data":"12620df851185bc029ca1460e403f68f2d753350afa10fe2c7a9008c8d06def5"}
May 25 13:04:13.506715 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.506693    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pqbcw" event={"ID":"2b9ff95a-a2af-4375-beea-7e288a65ab11","Type":"ContainerStarted","Data":"c0ae59eb6f84b1d279ed1c08d71e26b93c40f9e7fa120d904d80367a6d86eb04"}
May 25 13:04:13.507460 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.507439    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6hzsz" event={"ID":"1363056f-5cf2-4736-8032-7cbc4cd602e1","Type":"ContainerStarted","Data":"bb5e5b46ae4a5089075335f4b5b39306c02ea6963b581f71d812b5b5750319d7"}
May 25 13:04:13.508719 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.508696    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8rwnv" event={"ID":"feaa61e8-ce75-4474-8a0c-bc149dcb8074","Type":"ContainerStarted","Data":"053a072bbb120985f50c6c5afaa883673851c6b23e9b65425d919462640da39f"}
May 25 13:04:13.509715 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.509673    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"54ead090d89e2bd5e7d5db832485477df079bc8c87a9ad6d24a761c820a1821a"}
May 25 13:04:13.516874 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.516806    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-55.ec2.internal" podStartSLOduration=2.516794956 podStartE2EDuration="2.516794956s" podCreationTimestamp="2026-05-25 13:04:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:04:13.51654235 +0000 UTC m=+3.555299479" watchObservedRunningTime="2026-05-25 13:04:13.516794956 +0000 UTC m=+3.555552080"
May 25 13:04:13.983669 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.983615    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:13.983669 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:13.983674    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:13.983933 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:13.983826    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:13.983933 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:13.983843    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:13.983933 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:13.983855    2549 projected.go:194] Error preparing data for projected volume kube-api-access-g2bj6 for pod openshift-network-diagnostics/network-check-target-qnwqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:13.983933 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:13.983916    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6 podName:10569d7e-c62f-4724-b920-daf76ac7365f nodeName:}" failed. No retries permitted until 2026-05-25 13:04:15.983898175 +0000 UTC m=+6.022655283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2bj6" (UniqueName: "kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6") pod "network-check-target-qnwqv" (UID: "10569d7e-c62f-4724-b920-daf76ac7365f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:13.984230 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:13.984154    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:13.984230 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:13.984206    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:15.984190977 +0000 UTC m=+6.022948083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:14.493976 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:14.493941    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:14.494460 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:14.494094    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:14.530590 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:14.530541    2549 generic.go:358] "Generic (PLEG): container finished" podID="085c1fe23cb3761f6985c2e4863c5818" containerID="1448adddf2e85fe1a83e7c1b7069dfae45c8ba6816a37212caa352bcefcc84b3" exitCode=0
May 25 13:04:14.531148 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:14.531100    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal" event={"ID":"085c1fe23cb3761f6985c2e4863c5818","Type":"ContainerDied","Data":"1448adddf2e85fe1a83e7c1b7069dfae45c8ba6816a37212caa352bcefcc84b3"}
May 25 13:04:15.492515 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:15.492370    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:15.492515 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:15.492494    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:15.543508 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:15.542815    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal" event={"ID":"085c1fe23cb3761f6985c2e4863c5818","Type":"ContainerStarted","Data":"b6c84bc77dab81a98f2c3cea7c7f75a0916f5e63c3f33e866ddd3fe51728d1c8"}
May 25 13:04:15.568620 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:15.568555    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-55.ec2.internal" podStartSLOduration=4.568532725 podStartE2EDuration="4.568532725s" podCreationTimestamp="2026-05-25 13:04:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:04:15.567013885 +0000 UTC m=+5.605771005" watchObservedRunningTime="2026-05-25 13:04:15.568532725 +0000 UTC m=+5.607289850"
May 25 13:04:16.002557 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:16.001718    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:16.002557 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:16.001777    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:16.002557 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:16.001941    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:16.002557 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:16.001964    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:16.002557 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:16.001977    2549 projected.go:194] Error preparing data for projected volume kube-api-access-g2bj6 for pod openshift-network-diagnostics/network-check-target-qnwqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:16.002557 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:16.002040    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6 podName:10569d7e-c62f-4724-b920-daf76ac7365f nodeName:}" failed. No retries permitted until 2026-05-25 13:04:20.002020491 +0000 UTC m=+10.040777596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2bj6" (UniqueName: "kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6") pod "network-check-target-qnwqv" (UID: "10569d7e-c62f-4724-b920-daf76ac7365f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:16.002557 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:16.002476    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:16.002557 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:16.002524    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:20.002508869 +0000 UTC m=+10.041265975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:16.492343 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:16.491819    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:16.492343 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:16.491960    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:17.492038 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:17.491561    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:17.492038 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:17.491686    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:18.492149 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:18.492114    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:18.492525 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:18.492284    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:19.491649 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:19.491613    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:19.491880 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:19.491853    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:20.039871 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:20.038989    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:20.039871 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:20.039055    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:20.039871 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:20.039239    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:20.039871 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:20.039257    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:20.039871 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:20.039290    2549 projected.go:194] Error preparing data for projected volume kube-api-access-g2bj6 for pod openshift-network-diagnostics/network-check-target-qnwqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:20.039871 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:20.039357    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6 podName:10569d7e-c62f-4724-b920-daf76ac7365f nodeName:}" failed. No retries permitted until 2026-05-25 13:04:28.039334373 +0000 UTC m=+18.078091500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2bj6" (UniqueName: "kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6") pod "network-check-target-qnwqv" (UID: "10569d7e-c62f-4724-b920-daf76ac7365f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:20.039871 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:20.039782    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:20.039871 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:20.039835    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:28.03981793 +0000 UTC m=+18.078575038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:20.502447 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:20.502395    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:20.502673 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:20.502647    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:21.492189 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:21.491756    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:21.492189 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:21.491859    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:22.492191 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:22.492151    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:22.492587 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:22.492318    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:23.491646 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:23.491613    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:23.491834 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:23.491723    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:24.492063 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:24.492027    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:24.492549 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:24.492166    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:25.267934 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.267892    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hrxhv"]
May 25 13:04:25.271004 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.270980    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.271146 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:25.271063    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:25.382084 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.382043    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-dbus\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.382247 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.382115    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.382247 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.382145    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-kubelet-config\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.483519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.483478    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-dbus\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.483692 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.483548    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.483692 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.483648    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-dbus\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.483692 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:25.483663    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:25.483818 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.483703    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-kubelet-config\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.483818 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:25.483726    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret podName:6ec74b53-fbd6-407f-a5c8-cefb6f613f10 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:25.983706394 +0000 UTC m=+16.022463510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret") pod "global-pull-secret-syncer-hrxhv" (UID: "6ec74b53-fbd6-407f-a5c8-cefb6f613f10") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:25.483818 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.483762    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-kubelet-config\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.491672 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.491637    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:25.491802 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:25.491736    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:25.988705 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:25.988664    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:25.989115 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:25.988805    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:25.989115 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:25.988866    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret podName:6ec74b53-fbd6-407f-a5c8-cefb6f613f10 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:26.988850648 +0000 UTC m=+17.027607770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret") pod "global-pull-secret-syncer-hrxhv" (UID: "6ec74b53-fbd6-407f-a5c8-cefb6f613f10") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:26.491916 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:26.491865    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:26.492092 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:26.492017    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:26.994878 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:26.994801    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:26.995342 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:26.994979    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:26.995342 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:26.995047    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret podName:6ec74b53-fbd6-407f-a5c8-cefb6f613f10 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:28.995029511 +0000 UTC m=+19.033786616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret") pod "global-pull-secret-syncer-hrxhv" (UID: "6ec74b53-fbd6-407f-a5c8-cefb6f613f10") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:27.492283 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:27.492228    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:27.492437 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:27.492232    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:27.492437 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:27.492373    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:27.492530 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:27.492480    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:28.103504 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:28.103465    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:28.103975 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:28.103524    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:28.103975 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:28.103619    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:28.103975 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:28.103665    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:28.103975 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:28.103683    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:28.103975 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:28.103697    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:44.103679173 +0000 UTC m=+34.142436297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:28.103975 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:28.103698    2549 projected.go:194] Error preparing data for projected volume kube-api-access-g2bj6 for pod openshift-network-diagnostics/network-check-target-qnwqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:28.103975 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:28.103756    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6 podName:10569d7e-c62f-4724-b920-daf76ac7365f nodeName:}" failed. No retries permitted until 2026-05-25 13:04:44.10373863 +0000 UTC m=+34.142495736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2bj6" (UniqueName: "kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6") pod "network-check-target-qnwqv" (UID: "10569d7e-c62f-4724-b920-daf76ac7365f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:28.491635 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:28.491598    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:28.491834 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:28.491749    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:29.009016 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:29.008977    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:29.009167 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:29.009121    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:29.009222 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:29.009175    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret podName:6ec74b53-fbd6-407f-a5c8-cefb6f613f10 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:33.009160657 +0000 UTC m=+23.047917759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret") pod "global-pull-secret-syncer-hrxhv" (UID: "6ec74b53-fbd6-407f-a5c8-cefb6f613f10") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:29.492255 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:29.492216    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:29.492703 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:29.492352    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:29.492703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:29.492221    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:29.492840 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:29.492776    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:30.493127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:30.493089    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:30.493595 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:30.493211    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:31.491620 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:31.491584    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:31.491620 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:31.491618    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:31.491846 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:31.491708    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:31.491846 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:31.491781    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:32.492464 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.492219    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:32.493323 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:32.492563    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:32.574375 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.574343    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8rwnv" event={"ID":"feaa61e8-ce75-4474-8a0c-bc149dcb8074","Type":"ContainerStarted","Data":"ee92baf935eb11a5b994625fc8b2f1046922b3f9fe62c061d2e2607695941d2f"}
May 25 13:04:32.576047 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.576024    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:04:32.576403 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.576376    2549 generic.go:358] "Generic (PLEG): container finished" podID="22897fc3-4085-4592-92b7-ba00a4a8dfaa" containerID="1ab0b78cf574dae08774c2f17bdb74f2f58ae619471eec52232c365aa24128a4" exitCode=1
May 25 13:04:32.576490 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.576455    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"37ad24be15ad40481e5502be77bef58838b49c05352b3705187a6838570af94a"}
May 25 13:04:32.576548 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.576493    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerDied","Data":"1ab0b78cf574dae08774c2f17bdb74f2f58ae619471eec52232c365aa24128a4"}
May 25 13:04:32.576548 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.576511    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"29f8f567f095115a37a317dd5d4516bc33372f7d3b3966add0972c6f2891023c"}
May 25 13:04:32.577700 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.577678    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mzsn" event={"ID":"8c6d0a9e-64c4-44de-8429-196cdbb51c51","Type":"ContainerStarted","Data":"f254f6964a48eeb2923a73b65788456f2e8905324e0df1207fed140e9cd5291a"}
May 25 13:04:32.578877 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.578858    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r" event={"ID":"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a","Type":"ContainerStarted","Data":"1848df56f03b9ccb9cdec47c8d26103a1b93afc30dbd70029cad251ebdb35b27"}
May 25 13:04:32.580202 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.580177    2549 generic.go:358] "Generic (PLEG): container finished" podID="218a9570-215b-4ca9-8d61-f9f39f8295e5" containerID="d302975fc86a946601588ec6664438bc2280bfc041962c3b3eaca0e2273de638" exitCode=0
May 25 13:04:32.580405 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.580236    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" event={"ID":"218a9570-215b-4ca9-8d61-f9f39f8295e5","Type":"ContainerDied","Data":"d302975fc86a946601588ec6664438bc2280bfc041962c3b3eaca0e2273de638"}
May 25 13:04:32.581609 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.581570    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xs46z" event={"ID":"5a934d1e-67ac-490f-b795-9203848f808b","Type":"ContainerStarted","Data":"39212c380e3cc692bfd2415a81c1698f6cee6a82e1444492e362be537dea96fb"}
May 25 13:04:32.582954 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.582933    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pqbcw" event={"ID":"2b9ff95a-a2af-4375-beea-7e288a65ab11","Type":"ContainerStarted","Data":"3f3e06c3754276932582c6bb194ade74c223c08ae7a4466fb304789fb6159da4"}
May 25 13:04:32.584230 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.584211    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6hzsz" event={"ID":"1363056f-5cf2-4736-8032-7cbc4cd602e1","Type":"ContainerStarted","Data":"f25faed0e1c56828e1f74f84934afceecea47bd1115e247d3d6c010b36ef40eb"}
May 25 13:04:32.602131 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.601987    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8rwnv" podStartSLOduration=3.952163522 podStartE2EDuration="22.601971388s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.31441237 +0000 UTC m=+3.353169473" lastFinishedPulling="2026-05-25 13:04:31.964220219 +0000 UTC m=+22.002977339" observedRunningTime="2026-05-25 13:04:32.601485107 +0000 UTC m=+22.640242231" watchObservedRunningTime="2026-05-25 13:04:32.601971388 +0000 UTC m=+22.640728513"
May 25 13:04:32.617019 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.616974    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xs46z" podStartSLOduration=3.9107471179999997 podStartE2EDuration="22.616961631s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.320941241 +0000 UTC m=+3.359698346" lastFinishedPulling="2026-05-25 13:04:32.027155741 +0000 UTC m=+22.065912859" observedRunningTime="2026-05-25 13:04:32.616714482 +0000 UTC m=+22.655471606" watchObservedRunningTime="2026-05-25 13:04:32.616961631 +0000 UTC m=+22.655718754"
May 25 13:04:32.632003 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.631959    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-2mzsn" podStartSLOduration=3.885005193 podStartE2EDuration="22.631944323s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.310174807 +0000 UTC m=+3.348931910" lastFinishedPulling="2026-05-25 13:04:32.057113924 +0000 UTC m=+22.095871040" observedRunningTime="2026-05-25 13:04:32.63179658 +0000 UTC m=+22.670553695" watchObservedRunningTime="2026-05-25 13:04:32.631944323 +0000 UTC m=+22.670701464"
May 25 13:04:32.652704 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:32.652659    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pqbcw" podStartSLOduration=4.009371005 podStartE2EDuration="22.652644084s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.320948455 +0000 UTC m=+3.359705562" lastFinishedPulling="2026-05-25 13:04:31.964221538 +0000 UTC m=+22.002978641" observedRunningTime="2026-05-25 13:04:32.652418909 +0000 UTC m=+22.691176033" watchObservedRunningTime="2026-05-25 13:04:32.652644084 +0000 UTC m=+22.691401208"
May 25 13:04:33.041175 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.041081    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:33.041354 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:33.041243    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:33.041354 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:33.041336    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret podName:6ec74b53-fbd6-407f-a5c8-cefb6f613f10 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:41.041319329 +0000 UTC m=+31.080076432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret") pod "global-pull-secret-syncer-hrxhv" (UID: "6ec74b53-fbd6-407f-a5c8-cefb6f613f10") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:33.492490 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.492252    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:33.492936 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.492295    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:33.492936 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:33.492585    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:33.492936 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:33.492674    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:33.533786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.533758    2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 13:04:33.587447 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.587375    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r" event={"ID":"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a","Type":"ContainerStarted","Data":"749b0eefe65ff4ab465ed3962f2f6be748d30ed0ff48f6e588a227af4637cce8"}
May 25 13:04:33.589962 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.589934    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:04:33.590375 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.590351    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"bda06fd66240d953b35793c6ff879b74e04877129493fdd865e70778e2766465"}
May 25 13:04:33.590435 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.590385    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"6b935a55b4c6026c6f0d5b1b8d06fec86d2f8f73fd48d6a74325b019e1818e1d"}
May 25 13:04:33.590435 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.590398    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"48f913a36b4fc0867ce6a9757d31e89b16b50dc49ba202334ae9522e13cbb4a9"}
May 25 13:04:33.591647 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.591625    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2hc5v" event={"ID":"93dafc65-1e53-4bf2-a21c-c8da486adaea","Type":"ContainerStarted","Data":"91cab444c82eb4cca57dc9f6dcd5dbe6f5f776a4442318a7a2b0dfb16951f50a"}
May 25 13:04:33.606472 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.606431    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-6hzsz" podStartSLOduration=4.897546549 podStartE2EDuration="23.606416267s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.320978684 +0000 UTC m=+3.359735787" lastFinishedPulling="2026-05-25 13:04:32.029848396 +0000 UTC m=+22.068605505" observedRunningTime="2026-05-25 13:04:32.741409113 +0000 UTC m=+22.780166236" watchObservedRunningTime="2026-05-25 13:04:33.606416267 +0000 UTC m=+23.645173391"
May 25 13:04:33.607061 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:33.607016    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2hc5v" podStartSLOduration=4.9529970599999995 podStartE2EDuration="23.607006672s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.310213692 +0000 UTC m=+3.348970794" lastFinishedPulling="2026-05-25 13:04:31.964223304 +0000 UTC m=+22.002980406" observedRunningTime="2026-05-25 13:04:33.60640541 +0000 UTC m=+23.645162539" watchObservedRunningTime="2026-05-25 13:04:33.607006672 +0000 UTC m=+23.645763797"
May 25 13:04:34.444924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:34.444800    2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T13:04:33.53378004Z","UUID":"79fead43-6e8b-46bc-89ad-9acc73b9f03f","Handler":null,"Name":"","Endpoint":""}
May 25 13:04:34.447192 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:34.447169    2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 13:04:34.447365 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:34.447199    2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 13:04:34.491724 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:34.491676    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:34.491881 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:34.491811    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:35.492264 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:35.492233    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:35.492911 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:35.492369    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:35.492911 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:35.492407    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:35.492911 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:35.492472    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:35.598349 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:35.598316    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:04:35.598730 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:35.598696    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"0ebab267ce5c0d9f7a48e184b92fe40aee4a70c4b3643bf78feb405ba91335ec"}
May 25 13:04:35.600649 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:35.600621    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r" event={"ID":"d7e58a46-c553-4bf1-96dc-ad68ddac1b4a","Type":"ContainerStarted","Data":"41b5109cd2dec384c07c0f27fc3f6050f5e22056ba2ca150bbbf1133cb1602cf"}
May 25 13:04:35.616362 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:35.616314    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8c6r" podStartSLOduration=4.394329855 podStartE2EDuration="25.616302195s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.310171876 +0000 UTC m=+3.348928978" lastFinishedPulling="2026-05-25 13:04:34.532144202 +0000 UTC m=+24.570901318" observedRunningTime="2026-05-25 13:04:35.616007585 +0000 UTC m=+25.654764710" watchObservedRunningTime="2026-05-25 13:04:35.616302195 +0000 UTC m=+25.655059319"
May 25 13:04:36.494139 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:36.491999    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:36.494139 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:36.492176    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:36.931007 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:36.930969    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:36.931661 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:36.931639    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:37.491969 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.491751    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:37.492113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.491750    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:37.492113 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:37.492005    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:37.492113 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:37.492091    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:37.607767 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.607737    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:04:37.608198 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.608063    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"b41013d14efb7c3f578495cd8a4ea202d80a8efb4d541c800873017303062075"}
May 25 13:04:37.608469 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.608442    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:37.608635 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.608481    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:37.608635 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.608493    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:37.608732 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.608644    2549 scope.go:117] "RemoveContainer" containerID="1ab0b78cf574dae08774c2f17bdb74f2f58ae619471eec52232c365aa24128a4"
May 25 13:04:37.609912 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.609885    2549 generic.go:358] "Generic (PLEG): container finished" podID="218a9570-215b-4ca9-8d61-f9f39f8295e5" containerID="fd8984200bfb8252e1dd91ac4bedacde8197d8faf215b132f11063ca803d2cb0" exitCode=0
May 25 13:04:37.610017 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.609924    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" event={"ID":"218a9570-215b-4ca9-8d61-f9f39f8295e5","Type":"ContainerDied","Data":"fd8984200bfb8252e1dd91ac4bedacde8197d8faf215b132f11063ca803d2cb0"}
May 25 13:04:37.610251 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.610225    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:37.614956 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.611298    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8rwnv"
May 25 13:04:37.631208 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.631187    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:37.631328 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:37.631303    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:04:38.491749 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:38.491722    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:38.491887 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:38.491827    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:38.613752 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:38.613716    2549 generic.go:358] "Generic (PLEG): container finished" podID="218a9570-215b-4ca9-8d61-f9f39f8295e5" containerID="e93f5467e646a8ad769f03747da2d4ebf16de8c641e98de4296c6727c1b62bd3" exitCode=0
May 25 13:04:38.614192 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:38.613762    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" event={"ID":"218a9570-215b-4ca9-8d61-f9f39f8295e5","Type":"ContainerDied","Data":"e93f5467e646a8ad769f03747da2d4ebf16de8c641e98de4296c6727c1b62bd3"}
May 25 13:04:38.617451 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:38.617433    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:04:38.617848 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:38.617786    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" event={"ID":"22897fc3-4085-4592-92b7-ba00a4a8dfaa","Type":"ContainerStarted","Data":"eb8a2add6a93a841f686a01428cfd40f1f3262dc42ea498bdad383da8074f0a2"}
May 25 13:04:38.660895 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:38.660807    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s" podStartSLOduration=9.843216584 podStartE2EDuration="28.660792965s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.278651643 +0000 UTC m=+3.317408745" lastFinishedPulling="2026-05-25 13:04:32.096228007 +0000 UTC m=+22.134985126" observedRunningTime="2026-05-25 13:04:38.659888007 +0000 UTC m=+28.698645164" watchObservedRunningTime="2026-05-25 13:04:38.660792965 +0000 UTC m=+28.699550089"
May 25 13:04:39.135996 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:39.135966    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrxhv"]
May 25 13:04:39.136161 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:39.136095    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:39.136221 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:39.136205    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:39.139426 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:39.139401    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kmndm"]
May 25 13:04:39.139558 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:39.139525    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:39.139653 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:39.139622    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:39.140043 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:39.140013    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qnwqv"]
May 25 13:04:39.140239 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:39.140123    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:39.140362 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:39.140332    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:39.621631 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:39.621600    2549 generic.go:358] "Generic (PLEG): container finished" podID="218a9570-215b-4ca9-8d61-f9f39f8295e5" containerID="436133df129c1e63f184bdd576806a679d2503910f4d90def35d876c985dab9d" exitCode=0
May 25 13:04:39.622020 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:39.621681    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" event={"ID":"218a9570-215b-4ca9-8d61-f9f39f8295e5","Type":"ContainerDied","Data":"436133df129c1e63f184bdd576806a679d2503910f4d90def35d876c985dab9d"}
May 25 13:04:40.494432 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:40.494399    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:40.494643 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:40.494399    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:40.494643 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:40.494520    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:40.494643 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:40.494399    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:40.494643 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:40.494600    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:40.494844 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:40.494673    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:41.103171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:41.103131    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:41.103624 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:41.103313    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 13:04:41.103624 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:41.103379    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret podName:6ec74b53-fbd6-407f-a5c8-cefb6f613f10 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:57.103363932 +0000 UTC m=+47.142121034 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret") pod "global-pull-secret-syncer-hrxhv" (UID: "6ec74b53-fbd6-407f-a5c8-cefb6f613f10") : object "kube-system"/"original-pull-secret" not registered
May 25 13:04:42.491877 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:42.491844    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:42.492550 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:42.491844    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:42.492550 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:42.491981    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:42.492550 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:42.491844    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:42.492550 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:42.492045    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:42.492550 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:42.492122    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:44.127938 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.127895    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:44.128517 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.127973    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:44.128517 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.128068    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:44.128517 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.128113    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:04:44.128517 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.128126    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:04:44.128517 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.128135    2549 projected.go:194] Error preparing data for projected volume kube-api-access-g2bj6 for pod openshift-network-diagnostics/network-check-target-qnwqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:44.128517 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.128156    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:16.128134028 +0000 UTC m=+66.166891152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:04:44.128517 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.128184    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6 podName:10569d7e-c62f-4724-b920-daf76ac7365f nodeName:}" failed. No retries permitted until 2026-05-25 13:05:16.128168506 +0000 UTC m=+66.166925611 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2bj6" (UniqueName: "kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6") pod "network-check-target-qnwqv" (UID: "10569d7e-c62f-4724-b920-daf76ac7365f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:04:44.492201 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.491958    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:44.492201 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.491988    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:44.492201 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.491963    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:44.492201 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.492099    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxhv" podUID="6ec74b53-fbd6-407f-a5c8-cefb6f613f10"
May 25 13:04:44.492201 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.492144    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qnwqv" podUID="10569d7e-c62f-4724-b920-daf76ac7365f"
May 25 13:04:44.492642 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:44.492251    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:04:44.807980 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.807950    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-55.ec2.internal" event="NodeReady"
May 25 13:04:44.808175 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.808112    2549 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 13:04:44.850145 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.850107    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54579ff64f-hvqgf"]
May 25 13:04:44.883116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.883077    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"]
May 25 13:04:44.883335 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.883309    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:44.885851 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.885827    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 13:04:44.885972 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.885831    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4x6vz\""
May 25 13:04:44.886190 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.886170    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 13:04:44.886296 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.886196    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 13:04:44.893417 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.893213    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 13:04:44.902497 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.902429    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"]
May 25 13:04:44.902637 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.902533    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:04:44.906633 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.906595    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 25 13:04:44.906877 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.906855    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 25 13:04:44.906982 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.906881    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 25 13:04:44.906982 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.906855    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bfdll\""
May 25 13:04:44.907688 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.907670    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 25 13:04:44.926262 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.926232    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"]
May 25 13:04:44.926755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.926730    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:44.929020 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.928997    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 25 13:04:44.946466 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.946442    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"]
May 25 13:04:44.946587 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.946473    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"]
May 25 13:04:44.946587 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.946486    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54579ff64f-hvqgf"]
May 25 13:04:44.946587 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.946498    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"]
May 25 13:04:44.946587 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.946513    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fwdbr"]
May 25 13:04:44.946736 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.946615    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:44.949176 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.949154    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 25 13:04:44.949574 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.949554    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 25 13:04:44.949718 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.949698    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 25 13:04:44.949982 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.949959    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 25 13:04:44.966844 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.966819    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cflbv"]
May 25 13:04:44.967032 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.967008    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:44.969469 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.969444    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 13:04:44.969693 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.969674    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 13:04:44.969798 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.969776    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pq8ln\""
May 25 13:04:44.969798 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.969792    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 13:04:44.983008 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.982985    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cflbv"]
May 25 13:04:44.983113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.983013    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fwdbr"]
May 25 13:04:44.983153 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.983137    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cflbv"
May 25 13:04:44.986053 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.985775    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 13:04:44.986053 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.985885    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mbxnr\""
May 25 13:04:44.986704 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:44.986683    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 13:04:45.034492 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034406    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-certificates\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.034492 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034453    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-trusted-ca\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.034492 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034471    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-installation-pull-secrets\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.034492 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034490    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034517    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02005501-8898-4b7b-89aa-9ecd5eb8f788-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54c778cb7b-9vmh5\" (UID: \"02005501-8898-4b7b-89aa-9ecd5eb8f788\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034570    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-hub\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034600    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034639    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797sw\" (UniqueName: \"kubernetes.io/projected/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-kube-api-access-797sw\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034676    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-image-registry-private-configuration\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034697    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-tmp\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034712    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7lzd\" (UniqueName: \"kubernetes.io/projected/02005501-8898-4b7b-89aa-9ecd5eb8f788-kube-api-access-p7lzd\") pod \"managed-serviceaccount-addon-agent-54c778cb7b-9vmh5\" (UID: \"02005501-8898-4b7b-89aa-9ecd5eb8f788\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034742    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhx7b\" (UniqueName: \"kubernetes.io/projected/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-kube-api-access-lhx7b\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034761    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.034778 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034776    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6p7\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-kube-api-access-9x6p7\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.035060 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034793    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.035060 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034821    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.035060 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034837    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01f44ebc-24b7-48ea-a601-a0114de04f96-ca-trust-extracted\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.035060 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034851    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-bound-sa-token\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.035060 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.034873    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-ca\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.135779 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.135738    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.135802    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-797sw\" (UniqueName: \"kubernetes.io/projected/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-kube-api-access-797sw\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.135838    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-image-registry-private-configuration\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.135866    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-tmp\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.135977    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p7lzd\" (UniqueName: \"kubernetes.io/projected/02005501-8898-4b7b-89aa-9ecd5eb8f788-kube-api-access-p7lzd\") pod \"managed-serviceaccount-addon-agent-54c778cb7b-9vmh5\" (UID: \"02005501-8898-4b7b-89aa-9ecd5eb8f788\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136019    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwz2f\" (UniqueName: \"kubernetes.io/projected/0e7829ab-3942-479d-9ddf-12c4352e88d7-kube-api-access-wwz2f\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136071    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhx7b\" (UniqueName: \"kubernetes.io/projected/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-kube-api-access-lhx7b\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136099    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136128    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6p7\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-kube-api-access-9x6p7\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136152    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136175    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136237    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-tmp\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136246    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136293    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01f44ebc-24b7-48ea-a601-a0114de04f96-ca-trust-extracted\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136317    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-bound-sa-token\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.136345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136345    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-ca\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136370    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136395    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7829ab-3942-479d-9ddf-12c4352e88d7-config-volume\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136478    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-certificates\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.136477    2549 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.136503    2549 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54579ff64f-hvqgf: secret "image-registry-tls" not found
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136525    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-trusted-ca\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136554    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-installation-pull-secrets\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.136572    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls podName:01f44ebc-24b7-48ea-a601-a0114de04f96 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:45.636551604 +0000 UTC m=+35.675308719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls") pod "image-registry-54579ff64f-hvqgf" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96") : secret "image-registry-tls" not found
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.136728    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01f44ebc-24b7-48ea-a601-a0114de04f96-ca-trust-extracted\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.138238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.137457    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-certificates\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.138793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.138420    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-trusted-ca\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.138793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.138466    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.138793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.138499    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsr8\" (UniqueName: \"kubernetes.io/projected/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-kube-api-access-jrsr8\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:45.138793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.138526    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7829ab-3942-479d-9ddf-12c4352e88d7-tmp-dir\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.138793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.138559    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02005501-8898-4b7b-89aa-9ecd5eb8f788-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54c778cb7b-9vmh5\" (UID: \"02005501-8898-4b7b-89aa-9ecd5eb8f788\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:04:45.138793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.138581    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-hub\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.139732 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.139548    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.141116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.141092    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-ca\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.141219 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.141179    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-installation-pull-secrets\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.141792 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.141769    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.141897 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.141825    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-image-registry-private-configuration\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.142226 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.142181    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.142903 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.142881    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02005501-8898-4b7b-89aa-9ecd5eb8f788-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54c778cb7b-9vmh5\" (UID: \"02005501-8898-4b7b-89aa-9ecd5eb8f788\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:04:45.143134 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.143112    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.143134 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.143131    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-hub\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.145698 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.145676    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7lzd\" (UniqueName: \"kubernetes.io/projected/02005501-8898-4b7b-89aa-9ecd5eb8f788-kube-api-access-p7lzd\") pod \"managed-serviceaccount-addon-agent-54c778cb7b-9vmh5\" (UID: \"02005501-8898-4b7b-89aa-9ecd5eb8f788\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:04:45.146764 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.146738    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6p7\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-kube-api-access-9x6p7\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.146982 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.146880    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhx7b\" (UniqueName: \"kubernetes.io/projected/8b2b3506-2ff8-4cac-b83c-3625a228ddf5-kube-api-access-lhx7b\") pod \"klusterlet-addon-workmgr-5bf7cd7579-dz24g\" (UID: \"8b2b3506-2ff8-4cac-b83c-3625a228ddf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.147452 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.147430    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-bound-sa-token\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.148115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.148094    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-797sw\" (UniqueName: \"kubernetes.io/projected/e322a7ad-3eb5-494f-93ee-a6d76f5aebb6-kube-api-access-797sw\") pod \"cluster-proxy-proxy-agent-658684d9f9-v827v\" (UID: \"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.220307 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.220250    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:04:45.239075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239047    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsr8\" (UniqueName: \"kubernetes.io/projected/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-kube-api-access-jrsr8\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:45.239227 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239087    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7829ab-3942-479d-9ddf-12c4352e88d7-tmp-dir\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.239227 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239129    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2f\" (UniqueName: \"kubernetes.io/projected/0e7829ab-3942-479d-9ddf-12c4352e88d7-kube-api-access-wwz2f\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.239227 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239179    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.239399 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.239296    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:04:45.239399 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.239350    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls podName:0e7829ab-3942-479d-9ddf-12c4352e88d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:45.739329687 +0000 UTC m=+35.778086792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls") pod "dns-default-cflbv" (UID: "0e7829ab-3942-479d-9ddf-12c4352e88d7") : secret "dns-default-metrics-tls" not found
May 25 13:04:45.239399 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239375    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:45.239539 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239411    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7829ab-3942-479d-9ddf-12c4352e88d7-config-volume\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.239539 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.239494    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:04:45.239539 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239505    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7829ab-3942-479d-9ddf-12c4352e88d7-tmp-dir\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.239539 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.239529    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert podName:386b85a9-b8f6-4f1d-b0c8-e45da5a1b514 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:45.739520154 +0000 UTC m=+35.778277257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert") pod "ingress-canary-fwdbr" (UID: "386b85a9-b8f6-4f1d-b0c8-e45da5a1b514") : secret "canary-serving-cert" not found
May 25 13:04:45.239833 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239817    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7829ab-3942-479d-9ddf-12c4352e88d7-config-volume\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.239922 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.239901    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:45.251000 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.250954    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz2f\" (UniqueName: \"kubernetes.io/projected/0e7829ab-3942-479d-9ddf-12c4352e88d7-kube-api-access-wwz2f\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.251000 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.250972    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsr8\" (UniqueName: \"kubernetes.io/projected/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-kube-api-access-jrsr8\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:45.256317 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.256294    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:04:45.642949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.642910    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:45.643139 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.643090    2549 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:04:45.643139 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.643111    2549 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54579ff64f-hvqgf: secret "image-registry-tls" not found
May 25 13:04:45.643241 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.643186    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls podName:01f44ebc-24b7-48ea-a601-a0114de04f96 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:46.643164418 +0000 UTC m=+36.681921526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls") pod "image-registry-54579ff64f-hvqgf" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96") : secret "image-registry-tls" not found
May 25 13:04:45.744215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.744175    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:45.744215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:45.744223    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:45.744478 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.744340    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:04:45.744478 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.744353    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:04:45.744478 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.744400    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert podName:386b85a9-b8f6-4f1d-b0c8-e45da5a1b514 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:46.744382528 +0000 UTC m=+36.783139631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert") pod "ingress-canary-fwdbr" (UID: "386b85a9-b8f6-4f1d-b0c8-e45da5a1b514") : secret "canary-serving-cert" not found
May 25 13:04:45.744478 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:45.744421    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls podName:0e7829ab-3942-479d-9ddf-12c4352e88d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:46.744412258 +0000 UTC m=+36.783169379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls") pod "dns-default-cflbv" (UID: "0e7829ab-3942-479d-9ddf-12c4352e88d7") : secret "dns-default-metrics-tls" not found
May 25 13:04:46.491984 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.491940    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:04:46.492445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.491991    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:04:46.492445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.492175    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:46.494726 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.494705    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:04:46.495419 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.495399    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:04:46.495533 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.495446    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6pf86\""
May 25 13:04:46.495533 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.495456    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:04:46.495651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.495583    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 13:04:46.495705 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.495690    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tfm74\""
May 25 13:04:46.653955 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.652613    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:46.653955 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:46.652771    2549 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:04:46.653955 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:46.652788    2549 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54579ff64f-hvqgf: secret "image-registry-tls" not found
May 25 13:04:46.653955 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:46.652883    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls podName:01f44ebc-24b7-48ea-a601-a0114de04f96 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:48.652864092 +0000 UTC m=+38.691621217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls") pod "image-registry-54579ff64f-hvqgf" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96") : secret "image-registry-tls" not found
May 25 13:04:46.721866 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.721832    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"]
May 25 13:04:46.724876 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.724852    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"]
May 25 13:04:46.735753 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.735731    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"]
May 25 13:04:46.753255 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.753193    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:46.753255 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:46.753230    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:46.753454 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:46.753345    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:04:46.753454 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:46.753354    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:04:46.753454 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:46.753403    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert podName:386b85a9-b8f6-4f1d-b0c8-e45da5a1b514 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:48.753385455 +0000 UTC m=+38.792142572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert") pod "ingress-canary-fwdbr" (UID: "386b85a9-b8f6-4f1d-b0c8-e45da5a1b514") : secret "canary-serving-cert" not found
May 25 13:04:46.753454 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:46.753418    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls podName:0e7829ab-3942-479d-9ddf-12c4352e88d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:48.753411723 +0000 UTC m=+38.792168825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls") pod "dns-default-cflbv" (UID: "0e7829ab-3942-479d-9ddf-12c4352e88d7") : secret "dns-default-metrics-tls" not found
May 25 13:04:46.754719 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:46.754695    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02005501_8898_4b7b_89aa_9ecd5eb8f788.slice/crio-7ca7b77b1f91e67aff896b69cc40101addb610608a9116e100db09e0bc4a4d36 WatchSource:0}: Error finding container 7ca7b77b1f91e67aff896b69cc40101addb610608a9116e100db09e0bc4a4d36: Status 404 returned error can't find the container with id 7ca7b77b1f91e67aff896b69cc40101addb610608a9116e100db09e0bc4a4d36
May 25 13:04:46.755511 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:46.755489    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2b3506_2ff8_4cac_b83c_3625a228ddf5.slice/crio-a44342b5140cc3dd13806ac561d48b79b3ca852b37f442492693a6b31662c533 WatchSource:0}: Error finding container a44342b5140cc3dd13806ac561d48b79b3ca852b37f442492693a6b31662c533: Status 404 returned error can't find the container with id a44342b5140cc3dd13806ac561d48b79b3ca852b37f442492693a6b31662c533
May 25 13:04:46.756540 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:46.756513    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode322a7ad_3eb5_494f_93ee_a6d76f5aebb6.slice/crio-2c1f6771ba79007b91c9a02f009f18263e6a6102f4ac4e6f7f09c4245738578f WatchSource:0}: Error finding container 2c1f6771ba79007b91c9a02f009f18263e6a6102f4ac4e6f7f09c4245738578f: Status 404 returned error can't find the container with id 2c1f6771ba79007b91c9a02f009f18263e6a6102f4ac4e6f7f09c4245738578f
May 25 13:04:47.643772 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:47.643731    2549 generic.go:358] "Generic (PLEG): container finished" podID="218a9570-215b-4ca9-8d61-f9f39f8295e5" containerID="8305f4f0aa5f3d78014d9f0ec4e4305cf9fb04d67731a03ea0e462de5592a796" exitCode=0
May 25 13:04:47.644262 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:47.643812    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" event={"ID":"218a9570-215b-4ca9-8d61-f9f39f8295e5","Type":"ContainerDied","Data":"8305f4f0aa5f3d78014d9f0ec4e4305cf9fb04d67731a03ea0e462de5592a796"}
May 25 13:04:47.650337 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:47.650209    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g" event={"ID":"8b2b3506-2ff8-4cac-b83c-3625a228ddf5","Type":"ContainerStarted","Data":"a44342b5140cc3dd13806ac561d48b79b3ca852b37f442492693a6b31662c533"}
May 25 13:04:47.652720 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:47.652653    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" event={"ID":"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6","Type":"ContainerStarted","Data":"2c1f6771ba79007b91c9a02f009f18263e6a6102f4ac4e6f7f09c4245738578f"}
May 25 13:04:47.654147 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:47.654123    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5" event={"ID":"02005501-8898-4b7b-89aa-9ecd5eb8f788","Type":"ContainerStarted","Data":"7ca7b77b1f91e67aff896b69cc40101addb610608a9116e100db09e0bc4a4d36"}
May 25 13:04:48.663298 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:48.662381    2549 generic.go:358] "Generic (PLEG): container finished" podID="218a9570-215b-4ca9-8d61-f9f39f8295e5" containerID="fdd2687e92e4172d685bf6cf7ed3a1ad804d0c504bcf7067c443c1f2f7ce4b88" exitCode=0
May 25 13:04:48.663298 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:48.662653    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" event={"ID":"218a9570-215b-4ca9-8d61-f9f39f8295e5","Type":"ContainerDied","Data":"fdd2687e92e4172d685bf6cf7ed3a1ad804d0c504bcf7067c443c1f2f7ce4b88"}
May 25 13:04:48.673514 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:48.673484    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:48.673692 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:48.673642    2549 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:04:48.673692 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:48.673658    2549 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54579ff64f-hvqgf: secret "image-registry-tls" not found
May 25 13:04:48.673806 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:48.673710    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls podName:01f44ebc-24b7-48ea-a601-a0114de04f96 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:52.673691337 +0000 UTC m=+42.712448448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls") pod "image-registry-54579ff64f-hvqgf" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96") : secret "image-registry-tls" not found
May 25 13:04:48.775218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:48.774105    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:48.775218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:48.774165    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:48.775218 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:48.774583    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:04:48.775218 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:48.774667    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls podName:0e7829ab-3942-479d-9ddf-12c4352e88d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:52.774644282 +0000 UTC m=+42.813401384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls") pod "dns-default-cflbv" (UID: "0e7829ab-3942-479d-9ddf-12c4352e88d7") : secret "dns-default-metrics-tls" not found
May 25 13:04:48.775218 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:48.774743    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:04:48.775218 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:48.774807    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert podName:386b85a9-b8f6-4f1d-b0c8-e45da5a1b514 nodeName:}" failed. No retries permitted until 2026-05-25 13:04:52.774789377 +0000 UTC m=+42.813546494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert") pod "ingress-canary-fwdbr" (UID: "386b85a9-b8f6-4f1d-b0c8-e45da5a1b514") : secret "canary-serving-cert" not found
May 25 13:04:52.710285 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:52.710235    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:04:52.710791 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:52.710397    2549 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:04:52.710791 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:52.710418    2549 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54579ff64f-hvqgf: secret "image-registry-tls" not found
May 25 13:04:52.710791 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:52.710475    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls podName:01f44ebc-24b7-48ea-a601-a0114de04f96 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:00.710460474 +0000 UTC m=+50.749217576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls") pod "image-registry-54579ff64f-hvqgf" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96") : secret "image-registry-tls" not found
May 25 13:04:52.811399 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:52.811360    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:04:52.811579 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:52.811411    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:04:52.811579 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:52.811512    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:04:52.811579 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:52.811528    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:04:52.811579 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:52.811573    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert podName:386b85a9-b8f6-4f1d-b0c8-e45da5a1b514 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:00.81155418 +0000 UTC m=+50.850311301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert") pod "ingress-canary-fwdbr" (UID: "386b85a9-b8f6-4f1d-b0c8-e45da5a1b514") : secret "canary-serving-cert" not found
May 25 13:04:52.811805 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:04:52.811600    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls podName:0e7829ab-3942-479d-9ddf-12c4352e88d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:00.81158269 +0000 UTC m=+50.850339793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls") pod "dns-default-cflbv" (UID: "0e7829ab-3942-479d-9ddf-12c4352e88d7") : secret "dns-default-metrics-tls" not found
May 25 13:04:54.676003 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.675909    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5" event={"ID":"02005501-8898-4b7b-89aa-9ecd5eb8f788","Type":"ContainerStarted","Data":"ee0817681a6be7ca66640fccfc0c8d80cad16ecf2ad78e4fc31f93ddb895e231"}
May 25 13:04:54.678772 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.678731    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" event={"ID":"218a9570-215b-4ca9-8d61-f9f39f8295e5","Type":"ContainerStarted","Data":"e165e95f9a101303efca33d6d17b4f0f77ba364e5003d30b38a2a69cf79ea146"}
May 25 13:04:54.680035 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.680013    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g" event={"ID":"8b2b3506-2ff8-4cac-b83c-3625a228ddf5","Type":"ContainerStarted","Data":"d8f67f85e97d0bdfab3ca9a8ea3972b987c451d0723db3858b5a1a6bf242a1a3"}
May 25 13:04:54.680185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.680172    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:54.681316 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.681295    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" event={"ID":"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6","Type":"ContainerStarted","Data":"f753c674172d120a88b6b0b9c78e04a1f2a40c68f8b177a9c42ad88eb7635a5c"}
May 25 13:04:54.681987 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.681971    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:04:54.690756 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.690722    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5" podStartSLOduration=17.050623658 podStartE2EDuration="24.690711157s" podCreationTimestamp="2026-05-25 13:04:30 +0000 UTC" firstStartedPulling="2026-05-25 13:04:46.764093918 +0000 UTC m=+36.802851019" lastFinishedPulling="2026-05-25 13:04:54.404181399 +0000 UTC m=+44.442938518" observedRunningTime="2026-05-25 13:04:54.689952882 +0000 UTC m=+44.728710007" watchObservedRunningTime="2026-05-25 13:04:54.690711157 +0000 UTC m=+44.729468307"
May 25 13:04:54.710031 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.709988    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lw2vp" podStartSLOduration=11.205046304 podStartE2EDuration="44.709960016s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:04:13.280687448 +0000 UTC m=+3.319444564" lastFinishedPulling="2026-05-25 13:04:46.785601173 +0000 UTC m=+36.824358276" observedRunningTime="2026-05-25 13:04:54.709321343 +0000 UTC m=+44.748078468" watchObservedRunningTime="2026-05-25 13:04:54.709960016 +0000 UTC m=+44.748717140"
May 25 13:04:54.724667 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:54.724619    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g" podStartSLOduration=17.084304818 podStartE2EDuration="24.72460535s" podCreationTimestamp="2026-05-25 13:04:30 +0000 UTC" firstStartedPulling="2026-05-25 13:04:46.763878349 +0000 UTC m=+36.802635461" lastFinishedPulling="2026-05-25 13:04:54.404178878 +0000 UTC m=+44.442935993" observedRunningTime="2026-05-25 13:04:54.724463472 +0000 UTC m=+44.763220596" watchObservedRunningTime="2026-05-25 13:04:54.72460535 +0000 UTC m=+44.763362473"
May 25 13:04:57.147222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:57.147179    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:57.151319 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:57.151263    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ec74b53-fbd6-407f-a5c8-cefb6f613f10-original-pull-secret\") pod \"global-pull-secret-syncer-hrxhv\" (UID: \"6ec74b53-fbd6-407f-a5c8-cefb6f613f10\") " pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:57.315326 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:57.315284    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxhv"
May 25 13:04:57.708769 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:57.708628    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrxhv"]
May 25 13:04:57.711076 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:04:57.711041    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec74b53_fbd6_407f_a5c8_cefb6f613f10.slice/crio-d92e3be6190c9a03e886e64a8fbd3d8467573f710021eccfa74de43e89b55148 WatchSource:0}: Error finding container d92e3be6190c9a03e886e64a8fbd3d8467573f710021eccfa74de43e89b55148: Status 404 returned error can't find the container with id d92e3be6190c9a03e886e64a8fbd3d8467573f710021eccfa74de43e89b55148
May 25 13:04:58.691750 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:58.691717    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrxhv" event={"ID":"6ec74b53-fbd6-407f-a5c8-cefb6f613f10","Type":"ContainerStarted","Data":"d92e3be6190c9a03e886e64a8fbd3d8467573f710021eccfa74de43e89b55148"}
May 25 13:04:58.694121 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:58.694085    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" event={"ID":"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6","Type":"ContainerStarted","Data":"7b57a28987b29b0ba0ff60cbfac73b8efb0ba36812f4300eb7a676f921e69a65"}
May 25 13:04:58.694235 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:04:58.694132    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" event={"ID":"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6","Type":"ContainerStarted","Data":"a5f749c5e66d2be29ee18b471f97864e67b0e30ea64dc185bba3995916fa7678"}
May 25 13:05:00.513537 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:00.513483    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" podStartSLOduration=19.277976877 podStartE2EDuration="30.513464835s" podCreationTimestamp="2026-05-25 13:04:30 +0000 UTC" firstStartedPulling="2026-05-25 13:04:46.763887837 +0000 UTC m=+36.802644950" lastFinishedPulling="2026-05-25 13:04:57.999375802 +0000 UTC m=+48.038132908" observedRunningTime="2026-05-25 13:04:58.712143369 +0000 UTC m=+48.750900493" watchObservedRunningTime="2026-05-25 13:05:00.513464835 +0000 UTC m=+50.552222008"
May 25 13:05:00.780610 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:00.780517    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:05:00.780771 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:00.780677    2549 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:00.780771 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:00.780699    2549 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54579ff64f-hvqgf: secret "image-registry-tls" not found
May 25 13:05:00.780771 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:00.780764    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls podName:01f44ebc-24b7-48ea-a601-a0114de04f96 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:16.780747343 +0000 UTC m=+66.819504444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls") pod "image-registry-54579ff64f-hvqgf" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96") : secret "image-registry-tls" not found
May 25 13:05:00.881456 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:00.881411    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:05:00.881636 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:00.881482    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:05:00.881636 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:00.881543    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:00.881636 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:00.881629    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls podName:0e7829ab-3942-479d-9ddf-12c4352e88d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:16.881604971 +0000 UTC m=+66.920362078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls") pod "dns-default-cflbv" (UID: "0e7829ab-3942-479d-9ddf-12c4352e88d7") : secret "dns-default-metrics-tls" not found
May 25 13:05:00.881798 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:00.881640    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:00.881798 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:00.881705    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert podName:386b85a9-b8f6-4f1d-b0c8-e45da5a1b514 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:16.881687066 +0000 UTC m=+66.920444172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert") pod "ingress-canary-fwdbr" (UID: "386b85a9-b8f6-4f1d-b0c8-e45da5a1b514") : secret "canary-serving-cert" not found
May 25 13:05:02.704696 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:02.704655    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrxhv" event={"ID":"6ec74b53-fbd6-407f-a5c8-cefb6f613f10","Type":"ContainerStarted","Data":"577ea096cd22b628072e770da38ae53561dd2a5dfa2403661b79971edd25938c"}
May 25 13:05:02.720131 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:02.720076    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hrxhv" podStartSLOduration=33.761935133 podStartE2EDuration="37.720062303s" podCreationTimestamp="2026-05-25 13:04:25 +0000 UTC" firstStartedPulling="2026-05-25 13:04:57.712633899 +0000 UTC m=+47.751391001" lastFinishedPulling="2026-05-25 13:05:01.67076107 +0000 UTC m=+51.709518171" observedRunningTime="2026-05-25 13:05:02.719340786 +0000 UTC m=+52.758097910" watchObservedRunningTime="2026-05-25 13:05:02.720062303 +0000 UTC m=+52.758819423"
May 25 13:05:09.632646 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:09.632616    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qx45s"
May 25 13:05:16.201662 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.201621    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:05:16.202061 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.201671    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:05:16.204059 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.204039    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:05:16.204108 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.204039    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:05:16.211816 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.211798    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 13:05:16.211862 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.211856    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:20.211840003 +0000 UTC m=+130.250597104 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : secret "metrics-daemon-secret" not found
May 25 13:05:16.213672 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.213658    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:05:16.224951 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.224915    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bj6\" (UniqueName: \"kubernetes.io/projected/10569d7e-c62f-4724-b920-daf76ac7365f-kube-api-access-g2bj6\") pod \"network-check-target-qnwqv\" (UID: \"10569d7e-c62f-4724-b920-daf76ac7365f\") " pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:05:16.504446 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.504369    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tfm74\""
May 25 13:05:16.512292 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.512258    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:05:16.631011 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.630980    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qnwqv"]
May 25 13:05:16.634052 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:05:16.634024    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10569d7e_c62f_4724_b920_daf76ac7365f.slice/crio-177928b4f63f1e115f8d688764e6161fb968761cb13f0a324b9fec2af9bd72e1 WatchSource:0}: Error finding container 177928b4f63f1e115f8d688764e6161fb968761cb13f0a324b9fec2af9bd72e1: Status 404 returned error can't find the container with id 177928b4f63f1e115f8d688764e6161fb968761cb13f0a324b9fec2af9bd72e1
May 25 13:05:16.739171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.739130    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qnwqv" event={"ID":"10569d7e-c62f-4724-b920-daf76ac7365f","Type":"ContainerStarted","Data":"177928b4f63f1e115f8d688764e6161fb968761cb13f0a324b9fec2af9bd72e1"}
May 25 13:05:16.807201 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.807109    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:05:16.807367 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.807215    2549 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:16.807367 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.807236    2549 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54579ff64f-hvqgf: secret "image-registry-tls" not found
May 25 13:05:16.807367 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.807318    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls podName:01f44ebc-24b7-48ea-a601-a0114de04f96 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:48.807296032 +0000 UTC m=+98.846053152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls") pod "image-registry-54579ff64f-hvqgf" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96") : secret "image-registry-tls" not found
May 25 13:05:16.907819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.907781    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:05:16.907819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:16.907826    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:05:16.908009 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.907925    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:16.908009 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.907932    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:16.908009 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.907980    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert podName:386b85a9-b8f6-4f1d-b0c8-e45da5a1b514 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:48.907965685 +0000 UTC m=+98.946722787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert") pod "ingress-canary-fwdbr" (UID: "386b85a9-b8f6-4f1d-b0c8-e45da5a1b514") : secret "canary-serving-cert" not found
May 25 13:05:16.908009 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:16.907995    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls podName:0e7829ab-3942-479d-9ddf-12c4352e88d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:48.907987283 +0000 UTC m=+98.946744385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls") pod "dns-default-cflbv" (UID: "0e7829ab-3942-479d-9ddf-12c4352e88d7") : secret "dns-default-metrics-tls" not found
May 25 13:05:20.749170 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:20.749130    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qnwqv" event={"ID":"10569d7e-c62f-4724-b920-daf76ac7365f","Type":"ContainerStarted","Data":"45a49e5111fa3966d343eab00fffc23aea9df13adc9452e9b351333f6cdd8350"}
May 25 13:05:20.749632 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:20.749295    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:05:20.769805 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:20.769757    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qnwqv" podStartSLOduration=67.324257972 podStartE2EDuration="1m10.769743095s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:05:16.636076789 +0000 UTC m=+66.674833890" lastFinishedPulling="2026-05-25 13:05:20.081561911 +0000 UTC m=+70.120319013" observedRunningTime="2026-05-25 13:05:20.7693018 +0000 UTC m=+70.808058926" watchObservedRunningTime="2026-05-25 13:05:20.769743095 +0000 UTC m=+70.808500260"
May 25 13:05:48.855140 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:48.855098    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:05:48.855568 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:48.855237    2549 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:05:48.855568 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:48.855260    2549 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54579ff64f-hvqgf: secret "image-registry-tls" not found
May 25 13:05:48.855568 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:48.855343    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls podName:01f44ebc-24b7-48ea-a601-a0114de04f96 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:52.855326872 +0000 UTC m=+162.894083978 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls") pod "image-registry-54579ff64f-hvqgf" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96") : secret "image-registry-tls" not found
May 25 13:05:48.955971 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:48.955894    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:05:48.956097 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:48.955981    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:05:48.956097 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:48.956029    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:48.956097 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:48.956058    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:48.956201 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:48.956100    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert podName:386b85a9-b8f6-4f1d-b0c8-e45da5a1b514 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:52.95608321 +0000 UTC m=+162.994840311 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert") pod "ingress-canary-fwdbr" (UID: "386b85a9-b8f6-4f1d-b0c8-e45da5a1b514") : secret "canary-serving-cert" not found
May 25 13:05:48.956201 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:05:48.956132    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls podName:0e7829ab-3942-479d-9ddf-12c4352e88d7 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:52.956120739 +0000 UTC m=+162.994877862 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls") pod "dns-default-cflbv" (UID: "0e7829ab-3942-479d-9ddf-12c4352e88d7") : secret "dns-default-metrics-tls" not found
May 25 13:05:51.754136 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:05:51.754105    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qnwqv"
May 25 13:06:18.406233 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:18.406203    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6hzsz_1363056f-5cf2-4736-8032-7cbc4cd602e1/dns-node-resolver/0.log"
May 25 13:06:19.408967 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:19.408940    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pqbcw_2b9ff95a-a2af-4375-beea-7e288a65ab11/node-ca/0.log"
May 25 13:06:20.289942 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:20.289903    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:06:20.290116 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:06:20.290046    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 13:06:20.290155 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:06:20.290125    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs podName:cf5b6105-22f6-4f92-9cd2-5078d464a323 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:22.290109008 +0000 UTC m=+252.328866110 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs") pod "network-metrics-daemon-kmndm" (UID: "cf5b6105-22f6-4f92-9cd2-5078d464a323") : secret "metrics-daemon-secret" not found
May 25 13:06:39.108494 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.108451    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dvqmv"]
May 25 13:06:39.112237 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.112211    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.114443 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.114420    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 13:06:39.114556 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.114479    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 13:06:39.115344 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.115325    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 13:06:39.115426 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.115383    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 13:06:39.115501 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.115483    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvggk\""
May 25 13:06:39.128925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.128898    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dvqmv"]
May 25 13:06:39.232548 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.232501    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25c33450-733b-42b2-bdaa-cf1c9734355d-data-volume\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.232767 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.232582    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25c33450-733b-42b2-bdaa-cf1c9734355d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.232767 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.232603    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25c33450-733b-42b2-bdaa-cf1c9734355d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.232767 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.232628    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25c33450-733b-42b2-bdaa-cf1c9734355d-crio-socket\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.232767 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.232658    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzmx\" (UniqueName: \"kubernetes.io/projected/25c33450-733b-42b2-bdaa-cf1c9734355d-kube-api-access-dtzmx\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.333621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.333577    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzmx\" (UniqueName: \"kubernetes.io/projected/25c33450-733b-42b2-bdaa-cf1c9734355d-kube-api-access-dtzmx\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.333786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.333684    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25c33450-733b-42b2-bdaa-cf1c9734355d-data-volume\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.333786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.333739    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25c33450-733b-42b2-bdaa-cf1c9734355d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.333891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.333864    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25c33450-733b-42b2-bdaa-cf1c9734355d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.333943 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.333924    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25c33450-733b-42b2-bdaa-cf1c9734355d-crio-socket\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.334034 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.334011    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25c33450-733b-42b2-bdaa-cf1c9734355d-crio-socket\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.334082 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.334042    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25c33450-733b-42b2-bdaa-cf1c9734355d-data-volume\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.334220 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.334204    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25c33450-733b-42b2-bdaa-cf1c9734355d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.336237 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.336217    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25c33450-733b-42b2-bdaa-cf1c9734355d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.343100 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.343068    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzmx\" (UniqueName: \"kubernetes.io/projected/25c33450-733b-42b2-bdaa-cf1c9734355d-kube-api-access-dtzmx\") pod \"insights-runtime-extractor-dvqmv\" (UID: \"25c33450-733b-42b2-bdaa-cf1c9734355d\") " pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.421722 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.421681    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dvqmv"
May 25 13:06:39.560857 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.560825    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dvqmv"]
May 25 13:06:39.564233 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:06:39.564205    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c33450_733b_42b2_bdaa_cf1c9734355d.slice/crio-fc5f872636b40e9024e58989b7ec6145f1b6fc5d3e7a987d5db0ceae4772547d WatchSource:0}: Error finding container fc5f872636b40e9024e58989b7ec6145f1b6fc5d3e7a987d5db0ceae4772547d: Status 404 returned error can't find the container with id fc5f872636b40e9024e58989b7ec6145f1b6fc5d3e7a987d5db0ceae4772547d
May 25 13:06:39.935841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.935807    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvqmv" event={"ID":"25c33450-733b-42b2-bdaa-cf1c9734355d","Type":"ContainerStarted","Data":"7b6674249e9e19bf642726f2945463ce960591cf9f4749df089e5f2975e16975"}
May 25 13:06:39.935841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:39.935844    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvqmv" event={"ID":"25c33450-733b-42b2-bdaa-cf1c9734355d","Type":"ContainerStarted","Data":"fc5f872636b40e9024e58989b7ec6145f1b6fc5d3e7a987d5db0ceae4772547d"}
May 25 13:06:40.939441 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:40.939407    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvqmv" event={"ID":"25c33450-733b-42b2-bdaa-cf1c9734355d","Type":"ContainerStarted","Data":"2b2c50716a38ef934c01af6a969c45b34553c25d85184fe4b51d5edbc17948f6"}
May 25 13:06:42.945629 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:42.945593    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvqmv" event={"ID":"25c33450-733b-42b2-bdaa-cf1c9734355d","Type":"ContainerStarted","Data":"363c9b9d57868d9c239e0d5db37996fecf1339723b0550a25b67860f2a2ec2a7"}
May 25 13:06:42.967133 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:42.967080    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dvqmv" podStartSLOduration=1.64381229 podStartE2EDuration="3.967067102s" podCreationTimestamp="2026-05-25 13:06:39 +0000 UTC" firstStartedPulling="2026-05-25 13:06:39.61929656 +0000 UTC m=+149.658053662" lastFinishedPulling="2026-05-25 13:06:41.942551357 +0000 UTC m=+151.981308474" observedRunningTime="2026-05-25 13:06:42.965764688 +0000 UTC m=+153.004521810" watchObservedRunningTime="2026-05-25 13:06:42.967067102 +0000 UTC m=+153.005824225"
May 25 13:06:47.898465 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:06:47.898399    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" podUID="01f44ebc-24b7-48ea-a601-a0114de04f96"
May 25 13:06:47.956492 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:47.956463    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:06:47.987033 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:06:47.986983    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-fwdbr" podUID="386b85a9-b8f6-4f1d-b0c8-e45da5a1b514"
May 25 13:06:47.994119 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:06:47.994091    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cflbv" podUID="0e7829ab-3942-479d-9ddf-12c4352e88d7"
May 25 13:06:48.554126 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.554087    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jqv4n"]
May 25 13:06:48.557363 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.557339    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.561634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.561607    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 13:06:48.561978 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.561956    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 13:06:48.562542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.562523    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 13:06:48.562753 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.562729    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 13:06:48.562849 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.562793    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 13:06:48.562849 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.562735    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vwldl\""
May 25 13:06:48.563193 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.563176    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 13:06:48.710077 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710040    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-root\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.710238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710106    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-wtmp\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.710238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710145    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.710238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710168    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.710238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710220    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49td\" (UniqueName: \"kubernetes.io/projected/f4517545-b70c-47d7-abfb-e1af57d3d3fb-kube-api-access-j49td\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.710415 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710307    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4517545-b70c-47d7-abfb-e1af57d3d3fb-metrics-client-ca\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.710415 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710359    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-sys\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.710415 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710407    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-tls\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.710507 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.710426    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-textfile\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811548 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811475    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-sys\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811548 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811538    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-tls\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811676 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811563    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-textfile\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811676 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811585    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-sys\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811676 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811600    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-root\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811676 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811639    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-root\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811676 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811667    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-wtmp\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811690    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811711    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811738    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j49td\" (UniqueName: \"kubernetes.io/projected/f4517545-b70c-47d7-abfb-e1af57d3d3fb-kube-api-access-j49td\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811781    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4517545-b70c-47d7-abfb-e1af57d3d3fb-metrics-client-ca\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.811891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.811825    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-wtmp\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.812114 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.812052    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-textfile\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.812396 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.812373    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.812455 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.812396    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4517545-b70c-47d7-abfb-e1af57d3d3fb-metrics-client-ca\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.814171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.814148    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.814331 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.814313    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f4517545-b70c-47d7-abfb-e1af57d3d3fb-node-exporter-tls\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.820462 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.820413    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49td\" (UniqueName: \"kubernetes.io/projected/f4517545-b70c-47d7-abfb-e1af57d3d3fb-kube-api-access-j49td\") pod \"node-exporter-jqv4n\" (UID: \"f4517545-b70c-47d7-abfb-e1af57d3d3fb\") " pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.866614 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.866575    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jqv4n"
May 25 13:06:48.875119 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:06:48.875080    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4517545_b70c_47d7_abfb_e1af57d3d3fb.slice/crio-3613c19ac290cc9fd1c63931ff2acae5e82dc54f219b433c3bb584c5a81fb5ef WatchSource:0}: Error finding container 3613c19ac290cc9fd1c63931ff2acae5e82dc54f219b433c3bb584c5a81fb5ef: Status 404 returned error can't find the container with id 3613c19ac290cc9fd1c63931ff2acae5e82dc54f219b433c3bb584c5a81fb5ef
May 25 13:06:48.959606 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.959561    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqv4n" event={"ID":"f4517545-b70c-47d7-abfb-e1af57d3d3fb","Type":"ContainerStarted","Data":"3613c19ac290cc9fd1c63931ff2acae5e82dc54f219b433c3bb584c5a81fb5ef"}
May 25 13:06:48.959606 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:48.959601    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:06:49.507783 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:06:49.507743    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kmndm" podUID="cf5b6105-22f6-4f92-9cd2-5078d464a323"
May 25 13:06:49.964326 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:49.964254    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqv4n" event={"ID":"f4517545-b70c-47d7-abfb-e1af57d3d3fb","Type":"ContainerStarted","Data":"0b3b46994f51c2226a27b96e5735abc449a0188b06b8849049e9a6022c558fef"}
May 25 13:06:50.967434 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:50.967400    2549 generic.go:358] "Generic (PLEG): container finished" podID="f4517545-b70c-47d7-abfb-e1af57d3d3fb" containerID="0b3b46994f51c2226a27b96e5735abc449a0188b06b8849049e9a6022c558fef" exitCode=0
May 25 13:06:50.967915 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:50.967453    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqv4n" event={"ID":"f4517545-b70c-47d7-abfb-e1af57d3d3fb","Type":"ContainerDied","Data":"0b3b46994f51c2226a27b96e5735abc449a0188b06b8849049e9a6022c558fef"}
May 25 13:06:51.975809 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:51.975769    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqv4n" event={"ID":"f4517545-b70c-47d7-abfb-e1af57d3d3fb","Type":"ContainerStarted","Data":"384e829fec4f1349706250545a7cc10b7632fc7ffdd60ae29a8b5f53e9abb2de"}
May 25 13:06:51.975809 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:51.975807    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqv4n" event={"ID":"f4517545-b70c-47d7-abfb-e1af57d3d3fb","Type":"ContainerStarted","Data":"6385127e089a283806a4bfee5c647c70eb78d4535861e4bb91696dd1bf9f0ef8"}
May 25 13:06:51.995391 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:51.995341    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jqv4n" podStartSLOduration=3.030953693 podStartE2EDuration="3.995325275s" podCreationTimestamp="2026-05-25 13:06:48 +0000 UTC" firstStartedPulling="2026-05-25 13:06:48.876793246 +0000 UTC m=+158.915550350" lastFinishedPulling="2026-05-25 13:06:49.841164831 +0000 UTC m=+159.879921932" observedRunningTime="2026-05-25 13:06:51.994577751 +0000 UTC m=+162.033334875" watchObservedRunningTime="2026-05-25 13:06:51.995325275 +0000 UTC m=+162.034082390"
May 25 13:06:52.949367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:52.949317    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:06:52.952029 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:52.952004    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"image-registry-54579ff64f-hvqgf\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") " pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:06:53.049924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.049878    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:06:53.050342 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.049940    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:06:53.052451 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.052425    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7829ab-3942-479d-9ddf-12c4352e88d7-metrics-tls\") pod \"dns-default-cflbv\" (UID: \"0e7829ab-3942-479d-9ddf-12c4352e88d7\") " pod="openshift-dns/dns-default-cflbv"
May 25 13:06:53.052559 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.052474    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/386b85a9-b8f6-4f1d-b0c8-e45da5a1b514-cert\") pod \"ingress-canary-fwdbr\" (UID: \"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514\") " pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:06:53.059683 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.059658    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4x6vz\""
May 25 13:06:53.067694 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.067668    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:06:53.163179 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.163151    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pq8ln\""
May 25 13:06:53.171135 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.171102    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fwdbr"
May 25 13:06:53.221236 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.221203    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54579ff64f-hvqgf"]
May 25 13:06:53.234911 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:06:53.234878    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f44ebc_24b7_48ea_a601_a0114de04f96.slice/crio-dc0e6e5bff6e962771587336a4de02e0e6f2072cb5e904afa6356c2ee8a7c053 WatchSource:0}: Error finding container dc0e6e5bff6e962771587336a4de02e0e6f2072cb5e904afa6356c2ee8a7c053: Status 404 returned error can't find the container with id dc0e6e5bff6e962771587336a4de02e0e6f2072cb5e904afa6356c2ee8a7c053
May 25 13:06:53.304084 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.304044    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fwdbr"]
May 25 13:06:53.306930 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:06:53.306903    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386b85a9_b8f6_4f1d_b0c8_e45da5a1b514.slice/crio-5f43e2433cf31d945f4a62bab3fe82806579926d76f0b92ea098d2ce7840f116 WatchSource:0}: Error finding container 5f43e2433cf31d945f4a62bab3fe82806579926d76f0b92ea098d2ce7840f116: Status 404 returned error can't find the container with id 5f43e2433cf31d945f4a62bab3fe82806579926d76f0b92ea098d2ce7840f116
May 25 13:06:53.982284 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.982240    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" event={"ID":"01f44ebc-24b7-48ea-a601-a0114de04f96","Type":"ContainerStarted","Data":"d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d"}
May 25 13:06:53.982459 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.982294    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" event={"ID":"01f44ebc-24b7-48ea-a601-a0114de04f96","Type":"ContainerStarted","Data":"dc0e6e5bff6e962771587336a4de02e0e6f2072cb5e904afa6356c2ee8a7c053"}
May 25 13:06:53.982459 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.982331    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:06:53.983496 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:53.983469    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fwdbr" event={"ID":"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514","Type":"ContainerStarted","Data":"5f43e2433cf31d945f4a62bab3fe82806579926d76f0b92ea098d2ce7840f116"}
May 25 13:06:54.002259 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:54.002211    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" podStartSLOduration=164.00219599 podStartE2EDuration="2m44.00219599s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:54.001486934 +0000 UTC m=+164.040244059" watchObservedRunningTime="2026-05-25 13:06:54.00219599 +0000 UTC m=+164.040953114"
May 25 13:06:54.988505 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:54.988402    2549 generic.go:358] "Generic (PLEG): container finished" podID="02005501-8898-4b7b-89aa-9ecd5eb8f788" containerID="ee0817681a6be7ca66640fccfc0c8d80cad16ecf2ad78e4fc31f93ddb895e231" exitCode=255
May 25 13:06:54.988505 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:54.988479    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5" event={"ID":"02005501-8898-4b7b-89aa-9ecd5eb8f788","Type":"ContainerDied","Data":"ee0817681a6be7ca66640fccfc0c8d80cad16ecf2ad78e4fc31f93ddb895e231"}
May 25 13:06:54.988991 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:54.988851    2549 scope.go:117] "RemoveContainer" containerID="ee0817681a6be7ca66640fccfc0c8d80cad16ecf2ad78e4fc31f93ddb895e231"
May 25 13:06:54.989986 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:54.989959    2549 generic.go:358] "Generic (PLEG): container finished" podID="8b2b3506-2ff8-4cac-b83c-3625a228ddf5" containerID="d8f67f85e97d0bdfab3ca9a8ea3972b987c451d0723db3858b5a1a6bf242a1a3" exitCode=1
May 25 13:06:54.990094 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:54.990030    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g" event={"ID":"8b2b3506-2ff8-4cac-b83c-3625a228ddf5","Type":"ContainerDied","Data":"d8f67f85e97d0bdfab3ca9a8ea3972b987c451d0723db3858b5a1a6bf242a1a3"}
May 25 13:06:54.990453 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:54.990430    2549 scope.go:117] "RemoveContainer" containerID="d8f67f85e97d0bdfab3ca9a8ea3972b987c451d0723db3858b5a1a6bf242a1a3"
May 25 13:06:54.991618 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:54.991543    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fwdbr" event={"ID":"386b85a9-b8f6-4f1d-b0c8-e45da5a1b514","Type":"ContainerStarted","Data":"ae19619b12b2c0f240cf6498569a183e323952e77397aaa297b00e241fe55043"}
May 25 13:06:55.220805 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:55.220766    2549 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5"
May 25 13:06:55.241003 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:55.240915    2549 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:06:55.995070 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:55.995022    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54c778cb7b-9vmh5" event={"ID":"02005501-8898-4b7b-89aa-9ecd5eb8f788","Type":"ContainerStarted","Data":"82e467ae44a0c502c009235911afa94ec0fd25aef6c882924848a2bce117319f"}
May 25 13:06:55.996655 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:55.996631    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g" event={"ID":"8b2b3506-2ff8-4cac-b83c-3625a228ddf5","Type":"ContainerStarted","Data":"5b7be42c601214c743bd3b6edb3d25179b00a1ddf6ed774cdd39f360fc08b309"}
May 25 13:06:55.996805 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:55.996787    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:06:55.997377 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:55.997360    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf7cd7579-dz24g"
May 25 13:06:56.012189 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:56.012149    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fwdbr" podStartSLOduration=130.504944842 podStartE2EDuration="2m12.012139206s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:06:53.308857225 +0000 UTC m=+163.347614326" lastFinishedPulling="2026-05-25 13:06:54.816051568 +0000 UTC m=+164.854808690" observedRunningTime="2026-05-25 13:06:55.048246039 +0000 UTC m=+165.087003164" watchObservedRunningTime="2026-05-25 13:06:56.012139206 +0000 UTC m=+166.050896329"
May 25 13:06:58.492166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:58.492130    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cflbv"
May 25 13:06:58.495152 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:58.495128    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mbxnr\""
May 25 13:06:58.503210 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:58.503190    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cflbv"
May 25 13:06:58.636687 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:58.636616    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cflbv"]
May 25 13:06:58.639538 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:06:58.639511    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7829ab_3942_479d_9ddf_12c4352e88d7.slice/crio-27311fa2bae9d8bef0583de1ce00a06d90e72e879b977b3e73a9ed72d7a51dd7 WatchSource:0}: Error finding container 27311fa2bae9d8bef0583de1ce00a06d90e72e879b977b3e73a9ed72d7a51dd7: Status 404 returned error can't find the container with id 27311fa2bae9d8bef0583de1ce00a06d90e72e879b977b3e73a9ed72d7a51dd7
May 25 13:06:59.004160 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:06:59.004127    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cflbv" event={"ID":"0e7829ab-3942-479d-9ddf-12c4352e88d7","Type":"ContainerStarted","Data":"27311fa2bae9d8bef0583de1ce00a06d90e72e879b977b3e73a9ed72d7a51dd7"}
May 25 13:07:00.495195 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:00.495123    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:07:01.011327 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:01.011292    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cflbv" event={"ID":"0e7829ab-3942-479d-9ddf-12c4352e88d7","Type":"ContainerStarted","Data":"80e0c907c7dc23559e5e0ddc97215ed554f9b3bf7c950e7b4c5c809a99311550"}
May 25 13:07:01.011327 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:01.011331    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cflbv" event={"ID":"0e7829ab-3942-479d-9ddf-12c4352e88d7","Type":"ContainerStarted","Data":"e1c6a20b6a48d112af64037482630a0c4c933745a90587a88997f7223d3c71b2"}
May 25 13:07:01.011540 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:01.011520    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cflbv"
May 25 13:07:01.030634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:01.030589    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-cflbv" podStartSLOduration=135.430998134 podStartE2EDuration="2m17.030573087s" podCreationTimestamp="2026-05-25 13:04:44 +0000 UTC" firstStartedPulling="2026-05-25 13:06:58.64139678 +0000 UTC m=+168.680153881" lastFinishedPulling="2026-05-25 13:07:00.240971729 +0000 UTC m=+170.279728834" observedRunningTime="2026-05-25 13:07:01.029443741 +0000 UTC m=+171.068200867" watchObservedRunningTime="2026-05-25 13:07:01.030573087 +0000 UTC m=+171.069330210"
May 25 13:07:11.017211 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:11.017173    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cflbv"
May 25 13:07:13.071628 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:13.071595    2549 patch_prober.go:28] interesting pod/image-registry-54579ff64f-hvqgf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 13:07:13.071975 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:13.071645    2549 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" podUID="01f44ebc-24b7-48ea-a601-a0114de04f96" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:07:14.996548 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:14.996519    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:07:15.258360 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:15.258255    2549 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" podUID="e322a7ad-3eb5-494f-93ee-a6d76f5aebb6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 13:07:25.257747 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:25.257702    2549 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" podUID="e322a7ad-3eb5-494f-93ee-a6d76f5aebb6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 13:07:29.575046 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:29.575008    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54579ff64f-hvqgf"]
May 25 13:07:35.257861 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:35.257824    2549 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" podUID="e322a7ad-3eb5-494f-93ee-a6d76f5aebb6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 13:07:35.258343 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:35.257901    2549 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v"
May 25 13:07:35.258558 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:35.258522    2549 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"7b57a28987b29b0ba0ff60cbfac73b8efb0ba36812f4300eb7a676f921e69a65"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 25 13:07:35.258640 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:35.258586    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" podUID="e322a7ad-3eb5-494f-93ee-a6d76f5aebb6" containerName="service-proxy" containerID="cri-o://7b57a28987b29b0ba0ff60cbfac73b8efb0ba36812f4300eb7a676f921e69a65" gracePeriod=30
May 25 13:07:36.100179 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:36.100139    2549 generic.go:358] "Generic (PLEG): container finished" podID="e322a7ad-3eb5-494f-93ee-a6d76f5aebb6" containerID="7b57a28987b29b0ba0ff60cbfac73b8efb0ba36812f4300eb7a676f921e69a65" exitCode=2
May 25 13:07:36.100407 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:36.100215    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" event={"ID":"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6","Type":"ContainerDied","Data":"7b57a28987b29b0ba0ff60cbfac73b8efb0ba36812f4300eb7a676f921e69a65"}
May 25 13:07:36.100407 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:36.100250    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-658684d9f9-v827v" event={"ID":"e322a7ad-3eb5-494f-93ee-a6d76f5aebb6","Type":"ContainerStarted","Data":"1b08126bddac9c96f2864cd2a0f595d2748254b0ce87fbde5e34c32a22aa4bd6"}
May 25 13:07:54.598342 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:54.598300    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" podUID="01f44ebc-24b7-48ea-a601-a0114de04f96" containerName="registry" containerID="cri-o://d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d" gracePeriod=30
May 25 13:07:54.993144 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:54.993093    2549 patch_prober.go:28] interesting pod/image-registry-54579ff64f-hvqgf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" start-of-body=
May 25 13:07:54.993369 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:54.993166    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" podUID="01f44ebc-24b7-48ea-a601-a0114de04f96" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused"
May 25 13:07:55.835120 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.835096    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:07:55.931085 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931042    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") pod \"01f44ebc-24b7-48ea-a601-a0114de04f96\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") "
May 25 13:07:55.931379 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931094    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-installation-pull-secrets\") pod \"01f44ebc-24b7-48ea-a601-a0114de04f96\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") "
May 25 13:07:55.931379 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931135    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-trusted-ca\") pod \"01f44ebc-24b7-48ea-a601-a0114de04f96\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") "
May 25 13:07:55.931379 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931296    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x6p7\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-kube-api-access-9x6p7\") pod \"01f44ebc-24b7-48ea-a601-a0114de04f96\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") "
May 25 13:07:55.931379 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931354    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-bound-sa-token\") pod \"01f44ebc-24b7-48ea-a601-a0114de04f96\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") "
May 25 13:07:55.931621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931381    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-certificates\") pod \"01f44ebc-24b7-48ea-a601-a0114de04f96\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") "
May 25 13:07:55.931621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931404    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01f44ebc-24b7-48ea-a601-a0114de04f96-ca-trust-extracted\") pod \"01f44ebc-24b7-48ea-a601-a0114de04f96\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") "
May 25 13:07:55.931621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931460    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-image-registry-private-configuration\") pod \"01f44ebc-24b7-48ea-a601-a0114de04f96\" (UID: \"01f44ebc-24b7-48ea-a601-a0114de04f96\") "
May 25 13:07:55.931621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931545    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "01f44ebc-24b7-48ea-a601-a0114de04f96" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:55.931850 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931786    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-trusted-ca\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:07:55.931966 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.931881    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "01f44ebc-24b7-48ea-a601-a0114de04f96" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:55.933636 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.933606    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "01f44ebc-24b7-48ea-a601-a0114de04f96" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:55.933735 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.933714    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "01f44ebc-24b7-48ea-a601-a0114de04f96" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:55.933903 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.933877    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "01f44ebc-24b7-48ea-a601-a0114de04f96" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:55.934101 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.934076    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-kube-api-access-9x6p7" (OuterVolumeSpecName: "kube-api-access-9x6p7") pod "01f44ebc-24b7-48ea-a601-a0114de04f96" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96"). InnerVolumeSpecName "kube-api-access-9x6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:55.934159 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.934126    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "01f44ebc-24b7-48ea-a601-a0114de04f96" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:55.940639 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:55.940609    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f44ebc-24b7-48ea-a601-a0114de04f96-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "01f44ebc-24b7-48ea-a601-a0114de04f96" (UID: "01f44ebc-24b7-48ea-a601-a0114de04f96"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:07:56.032176 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.032141    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x6p7\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-kube-api-access-9x6p7\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:07:56.032176 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.032171    2549 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-bound-sa-token\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:07:56.032176 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.032182    2549 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-certificates\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:07:56.032424 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.032190    2549 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01f44ebc-24b7-48ea-a601-a0114de04f96-ca-trust-extracted\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:07:56.032424 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.032201    2549 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-image-registry-private-configuration\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:07:56.032424 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.032210    2549 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01f44ebc-24b7-48ea-a601-a0114de04f96-registry-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:07:56.032424 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.032219    2549 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01f44ebc-24b7-48ea-a601-a0114de04f96-installation-pull-secrets\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:07:56.151383 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.151348    2549 generic.go:358] "Generic (PLEG): container finished" podID="01f44ebc-24b7-48ea-a601-a0114de04f96" containerID="d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d" exitCode=0
May 25 13:07:56.151561 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.151431    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" event={"ID":"01f44ebc-24b7-48ea-a601-a0114de04f96","Type":"ContainerDied","Data":"d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d"}
May 25 13:07:56.151561 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.151437    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf"
May 25 13:07:56.151561 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.151458    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54579ff64f-hvqgf" event={"ID":"01f44ebc-24b7-48ea-a601-a0114de04f96","Type":"ContainerDied","Data":"dc0e6e5bff6e962771587336a4de02e0e6f2072cb5e904afa6356c2ee8a7c053"}
May 25 13:07:56.151561 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.151476    2549 scope.go:117] "RemoveContainer" containerID="d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d"
May 25 13:07:56.159816 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.159798    2549 scope.go:117] "RemoveContainer" containerID="d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d"
May 25 13:07:56.160068 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:07:56.160044    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d\": container with ID starting with d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d not found: ID does not exist" containerID="d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d"
May 25 13:07:56.160131 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.160080    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d"} err="failed to get container status \"d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d\": rpc error: code = NotFound desc = could not find container \"d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d\": container with ID starting with d0783aebc3194e778df89f4f0e6db66060058b0a6e1b0515b0e9d8ea017a884d not found: ID does not exist"
May 25 13:07:56.170891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.170870    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54579ff64f-hvqgf"]
May 25 13:07:56.172892 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.172868    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54579ff64f-hvqgf"]
May 25 13:07:56.496095 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:07:56.496055    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01f44ebc-24b7-48ea-a601-a0114de04f96" path="/var/lib/kubelet/pods/01f44ebc-24b7-48ea-a601-a0114de04f96/volumes"
May 25 13:08:22.330246 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:22.330132    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:08:22.332516 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:22.332493    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5b6105-22f6-4f92-9cd2-5078d464a323-metrics-certs\") pod \"network-metrics-daemon-kmndm\" (UID: \"cf5b6105-22f6-4f92-9cd2-5078d464a323\") " pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:08:22.398390 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:22.398355    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6pf86\""
May 25 13:08:22.407166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:22.407130    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmndm"
May 25 13:08:22.532374 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:22.532341    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kmndm"]
May 25 13:08:22.535533 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:08:22.535504    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5b6105_22f6_4f92_9cd2_5078d464a323.slice/crio-b523429b5af601597e6489b778274ff7e173fa41856dccf18c490798f19feca8 WatchSource:0}: Error finding container b523429b5af601597e6489b778274ff7e173fa41856dccf18c490798f19feca8: Status 404 returned error can't find the container with id b523429b5af601597e6489b778274ff7e173fa41856dccf18c490798f19feca8
May 25 13:08:23.219948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:23.219904    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmndm" event={"ID":"cf5b6105-22f6-4f92-9cd2-5078d464a323","Type":"ContainerStarted","Data":"b523429b5af601597e6489b778274ff7e173fa41856dccf18c490798f19feca8"}
May 25 13:08:24.223768 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:24.223733    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmndm" event={"ID":"cf5b6105-22f6-4f92-9cd2-5078d464a323","Type":"ContainerStarted","Data":"17886344d984a0c5da44c5cb81828c3af7a18ac516c620bf4a70a8e24fca8262"}
May 25 13:08:24.223768 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:24.223771    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmndm" event={"ID":"cf5b6105-22f6-4f92-9cd2-5078d464a323","Type":"ContainerStarted","Data":"0b9ef235fc907612ad5938ab2b2f0263cf96726673a53da544a751fbfa5c4789"}
May 25 13:08:24.242089 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:08:24.242045    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kmndm" podStartSLOduration=253.149422536 podStartE2EDuration="4m14.242032398s" podCreationTimestamp="2026-05-25 13:04:10 +0000 UTC" firstStartedPulling="2026-05-25 13:08:22.537402954 +0000 UTC m=+252.576160059" lastFinishedPulling="2026-05-25 13:08:23.630012813 +0000 UTC m=+253.668769921" observedRunningTime="2026-05-25 13:08:24.241046381 +0000 UTC m=+254.279803504" watchObservedRunningTime="2026-05-25 13:08:24.242032398 +0000 UTC m=+254.280789521"
May 25 13:09:10.370602 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:09:10.370572    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:09:10.372882 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:09:10.372863    2549 kubelet.go:1628] "Image garbage collection succeeded"
May 25 13:09:10.373487 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:09:10.373465    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:14:10.388487 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:14:10.388459    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:14:10.388982 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:14:10.388625    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:17:55.573764 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.573729    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"]
May 25 13:17:55.574234 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.573952    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f44ebc-24b7-48ea-a601-a0114de04f96" containerName="registry"
May 25 13:17:55.574234 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.573964    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f44ebc-24b7-48ea-a601-a0114de04f96" containerName="registry"
May 25 13:17:55.574234 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.574007    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="01f44ebc-24b7-48ea-a601-a0114de04f96" containerName="registry"
May 25 13:17:55.576682 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.576666    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.579129 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.579104    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 25 13:17:55.579286 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.579147    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 13:17:55.579933 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.579917    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 13:17:55.579995 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.579923    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 13:17:55.579995 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.579922    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-x45td\""
May 25 13:17:55.580250 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.580233    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 13:17:55.586821 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.586797    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"]
May 25 13:17:55.671163 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.671131    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/85f4fd69-8727-4398-94ae-3d6adb6af875-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.671356 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.671186    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.671356 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.671257    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmc6\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-kube-api-access-skmc6\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.772416 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.772384    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-skmc6\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-kube-api-access-skmc6\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.772625 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.772427    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/85f4fd69-8727-4398-94ae-3d6adb6af875-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.772625 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.772469    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.772625 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:55.772551    2549 secret.go:281] references non-existent secret key: tls.crt
May 25 13:17:55.772625 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:55.772563    2549 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 13:17:55.772625 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:55.772581    2549 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf: references non-existent secret key: tls.crt
May 25 13:17:55.772625 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:55.772630    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates podName:85f4fd69-8727-4398-94ae-3d6adb6af875 nodeName:}" failed. No retries permitted until 2026-05-25 13:17:56.272615052 +0000 UTC m=+826.311372154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates") pod "keda-metrics-apiserver-7c9f485588-66fnf" (UID: "85f4fd69-8727-4398-94ae-3d6adb6af875") : references non-existent secret key: tls.crt
May 25 13:17:55.772930 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.772838    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/85f4fd69-8727-4398-94ae-3d6adb6af875-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.775153 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.775129    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-fdlrr"]
May 25 13:17:55.778057 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.778042    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:55.780407 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.780389    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 25 13:17:55.786951 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.786922    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-fdlrr"]
May 25 13:17:55.788960 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.788939    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmc6\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-kube-api-access-skmc6\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:55.873041 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.872991    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rs7\" (UniqueName: \"kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-kube-api-access-h6rs7\") pod \"keda-admission-cf49989db-fdlrr\" (UID: \"d0dea6f9-55ac-4747-9f46-a6633e265e81\") " pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:55.873203 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.873052    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-certificates\") pod \"keda-admission-cf49989db-fdlrr\" (UID: \"d0dea6f9-55ac-4747-9f46-a6633e265e81\") " pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:55.974161 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.974114    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rs7\" (UniqueName: \"kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-kube-api-access-h6rs7\") pod \"keda-admission-cf49989db-fdlrr\" (UID: \"d0dea6f9-55ac-4747-9f46-a6633e265e81\") " pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:55.974298 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.974169    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-certificates\") pod \"keda-admission-cf49989db-fdlrr\" (UID: \"d0dea6f9-55ac-4747-9f46-a6633e265e81\") " pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:55.974298 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:55.974292    2549 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
May 25 13:17:55.974363 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:55.974315    2549 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-fdlrr: secret "keda-admission-webhooks-certs" not found
May 25 13:17:55.974396 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:55.974374    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-certificates podName:d0dea6f9-55ac-4747-9f46-a6633e265e81 nodeName:}" failed. No retries permitted until 2026-05-25 13:17:56.474358682 +0000 UTC m=+826.513115785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-certificates") pod "keda-admission-cf49989db-fdlrr" (UID: "d0dea6f9-55ac-4747-9f46-a6633e265e81") : secret "keda-admission-webhooks-certs" not found
May 25 13:17:55.982704 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:55.982673    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rs7\" (UniqueName: \"kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-kube-api-access-h6rs7\") pod \"keda-admission-cf49989db-fdlrr\" (UID: \"d0dea6f9-55ac-4747-9f46-a6633e265e81\") " pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:56.276841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:56.276745    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:56.276986 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:56.276848    2549 secret.go:281] references non-existent secret key: tls.crt
May 25 13:17:56.276986 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:56.276861    2549 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 13:17:56.276986 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:56.276876    2549 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf: references non-existent secret key: tls.crt
May 25 13:17:56.276986 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:56.276924    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates podName:85f4fd69-8727-4398-94ae-3d6adb6af875 nodeName:}" failed. No retries permitted until 2026-05-25 13:17:57.276911153 +0000 UTC m=+827.315668255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates") pod "keda-metrics-apiserver-7c9f485588-66fnf" (UID: "85f4fd69-8727-4398-94ae-3d6adb6af875") : references non-existent secret key: tls.crt
May 25 13:17:56.479026 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:56.478984    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-certificates\") pod \"keda-admission-cf49989db-fdlrr\" (UID: \"d0dea6f9-55ac-4747-9f46-a6633e265e81\") " pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:56.481564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:56.481525    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d0dea6f9-55ac-4747-9f46-a6633e265e81-certificates\") pod \"keda-admission-cf49989db-fdlrr\" (UID: \"d0dea6f9-55ac-4747-9f46-a6633e265e81\") " pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:56.689093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:56.689059    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:56.817638 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:56.817606    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-fdlrr"]
May 25 13:17:56.820987 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:17:56.820961    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dea6f9_55ac_4747_9f46_a6633e265e81.slice/crio-292e471d0ff02b7ce52f7d0c3d3a82023eeb0aee5899a3f27f8a5cbd95d46932 WatchSource:0}: Error finding container 292e471d0ff02b7ce52f7d0c3d3a82023eeb0aee5899a3f27f8a5cbd95d46932: Status 404 returned error can't find the container with id 292e471d0ff02b7ce52f7d0c3d3a82023eeb0aee5899a3f27f8a5cbd95d46932
May 25 13:17:56.822201 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:56.822185    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:17:57.286047 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:57.286013    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:57.286220 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:57.286158    2549 secret.go:281] references non-existent secret key: tls.crt
May 25 13:17:57.286220 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:57.286175    2549 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 13:17:57.286220 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:57.286193    2549 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf: references non-existent secret key: tls.crt
May 25 13:17:57.286346 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:17:57.286248    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates podName:85f4fd69-8727-4398-94ae-3d6adb6af875 nodeName:}" failed. No retries permitted until 2026-05-25 13:17:59.28623174 +0000 UTC m=+829.324988847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates") pod "keda-metrics-apiserver-7c9f485588-66fnf" (UID: "85f4fd69-8727-4398-94ae-3d6adb6af875") : references non-existent secret key: tls.crt
May 25 13:17:57.675564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:57.675508    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-fdlrr" event={"ID":"d0dea6f9-55ac-4747-9f46-a6633e265e81","Type":"ContainerStarted","Data":"292e471d0ff02b7ce52f7d0c3d3a82023eeb0aee5899a3f27f8a5cbd95d46932"}
May 25 13:17:59.303019 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:59.302958    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:59.305729 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:59.305707    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85f4fd69-8727-4398-94ae-3d6adb6af875-certificates\") pod \"keda-metrics-apiserver-7c9f485588-66fnf\" (UID: \"85f4fd69-8727-4398-94ae-3d6adb6af875\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:59.486200 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:59.486102    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:17:59.609459 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:59.609424    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"]
May 25 13:17:59.612580 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:17:59.612554    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f4fd69_8727_4398_94ae_3d6adb6af875.slice/crio-7aece32c344a4e9716ff3f6447d6e27003a3e8e8e276cf23eac038c98f6755d1 WatchSource:0}: Error finding container 7aece32c344a4e9716ff3f6447d6e27003a3e8e8e276cf23eac038c98f6755d1: Status 404 returned error can't find the container with id 7aece32c344a4e9716ff3f6447d6e27003a3e8e8e276cf23eac038c98f6755d1
May 25 13:17:59.684329 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:59.684263    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-fdlrr" event={"ID":"d0dea6f9-55ac-4747-9f46-a6633e265e81","Type":"ContainerStarted","Data":"308733133da6db62702938e0ca96140b1e6534ba831c6926b4029921a7aef586"}
May 25 13:17:59.684510 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:59.684395    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:17:59.685316 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:59.685289    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf" event={"ID":"85f4fd69-8727-4398-94ae-3d6adb6af875","Type":"ContainerStarted","Data":"7aece32c344a4e9716ff3f6447d6e27003a3e8e8e276cf23eac038c98f6755d1"}
May 25 13:17:59.701428 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:17:59.701347    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-fdlrr" podStartSLOduration=2.343275513 podStartE2EDuration="4.701329501s" podCreationTimestamp="2026-05-25 13:17:55 +0000 UTC" firstStartedPulling="2026-05-25 13:17:56.822340295 +0000 UTC m=+826.861097397" lastFinishedPulling="2026-05-25 13:17:59.180394282 +0000 UTC m=+829.219151385" observedRunningTime="2026-05-25 13:17:59.700905565 +0000 UTC m=+829.739662688" watchObservedRunningTime="2026-05-25 13:17:59.701329501 +0000 UTC m=+829.740086620"
May 25 13:18:02.696534 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:18:02.696492    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf" event={"ID":"85f4fd69-8727-4398-94ae-3d6adb6af875","Type":"ContainerStarted","Data":"86d5a439f6a3f370bcc72c7363a377d12f7e35dca93c8b804f1543b24f1961d7"}
May 25 13:18:02.696985 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:18:02.696780    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:18:02.714343 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:18:02.714265    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf" podStartSLOduration=5.484950742 podStartE2EDuration="7.714248853s" podCreationTimestamp="2026-05-25 13:17:55 +0000 UTC" firstStartedPulling="2026-05-25 13:17:59.613923243 +0000 UTC m=+829.652680344" lastFinishedPulling="2026-05-25 13:18:01.84322135 +0000 UTC m=+831.881978455" observedRunningTime="2026-05-25 13:18:02.713687267 +0000 UTC m=+832.752444414" watchObservedRunningTime="2026-05-25 13:18:02.714248853 +0000 UTC m=+832.753005978"
May 25 13:18:13.704112 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:18:13.704074    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-66fnf"
May 25 13:18:20.690376 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:18:20.690343    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-fdlrr"
May 25 13:19:07.615945 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.615855    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-nvlll"]
May 25 13:19:07.617960 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.617929    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:07.635348 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.635321    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 25 13:19:07.635867 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.635852    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 13:19:07.636399 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.636385    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-np85m\""
May 25 13:19:07.636452 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.636421    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 13:19:07.650925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.650889    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-nvlll"]
May 25 13:19:07.699184 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.699141    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56rl\" (UniqueName: \"kubernetes.io/projected/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-kube-api-access-w56rl\") pod \"kserve-controller-manager-7cd9b78f5d-nvlll\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:07.699386 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.699198    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert\") pod \"kserve-controller-manager-7cd9b78f5d-nvlll\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:07.800148 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.800103    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert\") pod \"kserve-controller-manager-7cd9b78f5d-nvlll\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:07.800361 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.800190    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w56rl\" (UniqueName: \"kubernetes.io/projected/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-kube-api-access-w56rl\") pod \"kserve-controller-manager-7cd9b78f5d-nvlll\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:07.800361 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:19:07.800242    2549 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
May 25 13:19:07.800361 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:19:07.800336    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert podName:4bb6af28-0dcd-454f-9542-5b9c12e9f0e7 nodeName:}" failed. No retries permitted until 2026-05-25 13:19:08.300317554 +0000 UTC m=+898.339074656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert") pod "kserve-controller-manager-7cd9b78f5d-nvlll" (UID: "4bb6af28-0dcd-454f-9542-5b9c12e9f0e7") : secret "kserve-webhook-server-cert" not found
May 25 13:19:07.812206 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:07.812177    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56rl\" (UniqueName: \"kubernetes.io/projected/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-kube-api-access-w56rl\") pod \"kserve-controller-manager-7cd9b78f5d-nvlll\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:08.304490 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:08.304428    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert\") pod \"kserve-controller-manager-7cd9b78f5d-nvlll\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:08.306967 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:08.306941    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert\") pod \"kserve-controller-manager-7cd9b78f5d-nvlll\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") " pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:08.530135 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:08.530091    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:08.670295 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:08.670242    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-nvlll"]
May 25 13:19:08.872116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:08.872081    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll" event={"ID":"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7","Type":"ContainerStarted","Data":"c03ed72b53e1104343d70b54226b6f4cd71c04432c0dd076e3c7d1fb72bf6c1d"}
May 25 13:19:11.042046 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:11.042020    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:19:11.042428 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:11.042025    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:19:11.882145 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:11.882105    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll" event={"ID":"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7","Type":"ContainerStarted","Data":"058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668"}
May 25 13:19:11.882368 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:11.882260    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:11.899709 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:11.899647    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll" podStartSLOduration=2.5149743 podStartE2EDuration="4.899630851s" podCreationTimestamp="2026-05-25 13:19:07 +0000 UTC" firstStartedPulling="2026-05-25 13:19:08.673675099 +0000 UTC m=+898.712432208" lastFinishedPulling="2026-05-25 13:19:11.058331656 +0000 UTC m=+901.097088759" observedRunningTime="2026-05-25 13:19:11.898726392 +0000 UTC m=+901.937483517" watchObservedRunningTime="2026-05-25 13:19:11.899630851 +0000 UTC m=+901.938387974"
May 25 13:19:42.889839 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:42.889803    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:45.340791 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.340750    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-nvlll"]
May 25 13:19:45.341204 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.340984    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll" podUID="4bb6af28-0dcd-454f-9542-5b9c12e9f0e7" containerName="manager" containerID="cri-o://058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668" gracePeriod=10
May 25 13:19:45.574109 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.574084    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:45.672646 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.672602    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert\") pod \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") "
May 25 13:19:45.672823 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.672654    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w56rl\" (UniqueName: \"kubernetes.io/projected/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-kube-api-access-w56rl\") pod \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\" (UID: \"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7\") "
May 25 13:19:45.674909 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.674874    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert" (OuterVolumeSpecName: "cert") pod "4bb6af28-0dcd-454f-9542-5b9c12e9f0e7" (UID: "4bb6af28-0dcd-454f-9542-5b9c12e9f0e7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:19:45.675019 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.674876    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-kube-api-access-w56rl" (OuterVolumeSpecName: "kube-api-access-w56rl") pod "4bb6af28-0dcd-454f-9542-5b9c12e9f0e7" (UID: "4bb6af28-0dcd-454f-9542-5b9c12e9f0e7"). InnerVolumeSpecName "kube-api-access-w56rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:19:45.773755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.773715    2549 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-cert\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:19:45.773755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.773749    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56rl\" (UniqueName: \"kubernetes.io/projected/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7-kube-api-access-w56rl\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:19:45.971909 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.971810    2549 generic.go:358] "Generic (PLEG): container finished" podID="4bb6af28-0dcd-454f-9542-5b9c12e9f0e7" containerID="058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668" exitCode=0
May 25 13:19:45.971909 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.971857    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll" event={"ID":"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7","Type":"ContainerDied","Data":"058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668"}
May 25 13:19:45.971909 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.971908    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll" event={"ID":"4bb6af28-0dcd-454f-9542-5b9c12e9f0e7","Type":"ContainerDied","Data":"c03ed72b53e1104343d70b54226b6f4cd71c04432c0dd076e3c7d1fb72bf6c1d"}
May 25 13:19:45.972162 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.971928    2549 scope.go:117] "RemoveContainer" containerID="058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668"
May 25 13:19:45.972162 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.971872    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7cd9b78f5d-nvlll"
May 25 13:19:45.979746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.979726    2549 scope.go:117] "RemoveContainer" containerID="058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668"
May 25 13:19:45.980005 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:19:45.979991    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668\": container with ID starting with 058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668 not found: ID does not exist" containerID="058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668"
May 25 13:19:45.980043 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.980013    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668"} err="failed to get container status \"058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668\": rpc error: code = NotFound desc = could not find container \"058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668\": container with ID starting with 058bfa986e7157894e0e5e612f4c2a853d749f40dc5c852a50605aa24bc04668 not found: ID does not exist"
May 25 13:19:45.991221 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.991198    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-nvlll"]
May 25 13:19:45.995025 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:45.995004    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7cd9b78f5d-nvlll"]
May 25 13:19:46.495371 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:19:46.495341    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb6af28-0dcd-454f-9542-5b9c12e9f0e7" path="/var/lib/kubelet/pods/4bb6af28-0dcd-454f-9542-5b9c12e9f0e7/volumes"
May 25 13:20:21.118707 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.118632    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-6b6fw"]
May 25 13:20:21.119124 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.118888    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb6af28-0dcd-454f-9542-5b9c12e9f0e7" containerName="manager"
May 25 13:20:21.119124 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.118899    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb6af28-0dcd-454f-9542-5b9c12e9f0e7" containerName="manager"
May 25 13:20:21.119124 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.118947    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="4bb6af28-0dcd-454f-9542-5b9c12e9f0e7" containerName="manager"
May 25 13:20:21.122004 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.121985    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:21.125989 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.124683    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-n7mhg\""
May 25 13:20:21.125989 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.124759    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 13:20:21.125989 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.124919    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 13:20:21.125989 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.125066    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 13:20:21.131749 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.131725    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-6b6fw"]
May 25 13:20:21.221654 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.221613    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4gq\" (UniqueName: \"kubernetes.io/projected/da4a53a7-6605-4b9d-8d3e-828d646ad77c-kube-api-access-9b4gq\") pod \"model-serving-api-86f7b4b499-6b6fw\" (UID: \"da4a53a7-6605-4b9d-8d3e-828d646ad77c\") " pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:21.221826 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.221702    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4a53a7-6605-4b9d-8d3e-828d646ad77c-tls-certs\") pod \"model-serving-api-86f7b4b499-6b6fw\" (UID: \"da4a53a7-6605-4b9d-8d3e-828d646ad77c\") " pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:21.322228 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.322179    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4gq\" (UniqueName: \"kubernetes.io/projected/da4a53a7-6605-4b9d-8d3e-828d646ad77c-kube-api-access-9b4gq\") pod \"model-serving-api-86f7b4b499-6b6fw\" (UID: \"da4a53a7-6605-4b9d-8d3e-828d646ad77c\") " pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:21.322454 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.322243    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4a53a7-6605-4b9d-8d3e-828d646ad77c-tls-certs\") pod \"model-serving-api-86f7b4b499-6b6fw\" (UID: \"da4a53a7-6605-4b9d-8d3e-828d646ad77c\") " pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:21.324829 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.324797    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4a53a7-6605-4b9d-8d3e-828d646ad77c-tls-certs\") pod \"model-serving-api-86f7b4b499-6b6fw\" (UID: \"da4a53a7-6605-4b9d-8d3e-828d646ad77c\") " pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:21.331523 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.331496    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4gq\" (UniqueName: \"kubernetes.io/projected/da4a53a7-6605-4b9d-8d3e-828d646ad77c-kube-api-access-9b4gq\") pod \"model-serving-api-86f7b4b499-6b6fw\" (UID: \"da4a53a7-6605-4b9d-8d3e-828d646ad77c\") " pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:21.435437 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.435348    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:21.561178 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:21.561146    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-6b6fw"]
May 25 13:20:21.564945 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:20:21.564919    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4a53a7_6605_4b9d_8d3e_828d646ad77c.slice/crio-214544f8adc7bf8197c96fe62113a09fc716d8206565eefda7d62bb5498593da WatchSource:0}: Error finding container 214544f8adc7bf8197c96fe62113a09fc716d8206565eefda7d62bb5498593da: Status 404 returned error can't find the container with id 214544f8adc7bf8197c96fe62113a09fc716d8206565eefda7d62bb5498593da
May 25 13:20:22.065453 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:22.065408    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-6b6fw" event={"ID":"da4a53a7-6605-4b9d-8d3e-828d646ad77c","Type":"ContainerStarted","Data":"214544f8adc7bf8197c96fe62113a09fc716d8206565eefda7d62bb5498593da"}
May 25 13:20:24.071752 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:24.071669    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-6b6fw" event={"ID":"da4a53a7-6605-4b9d-8d3e-828d646ad77c","Type":"ContainerStarted","Data":"95fecc3260dc7210c6c89b2afa7028da5bdb8f99097038ece89a5d61b2cc3baf"}
May 25 13:20:24.072115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:24.071811    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:24.095090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:24.095039    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-6b6fw" podStartSLOduration=0.881821946 podStartE2EDuration="3.095027563s" podCreationTimestamp="2026-05-25 13:20:21 +0000 UTC" firstStartedPulling="2026-05-25 13:20:21.566540359 +0000 UTC m=+971.605297460" lastFinishedPulling="2026-05-25 13:20:23.779745971 +0000 UTC m=+973.818503077" observedRunningTime="2026-05-25 13:20:24.092855592 +0000 UTC m=+974.131612751" watchObservedRunningTime="2026-05-25 13:20:24.095027563 +0000 UTC m=+974.133784686"
May 25 13:20:35.080979 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:35.080948    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-6b6fw"
May 25 13:20:54.595297 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.595230    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-4w4z2"]
May 25 13:20:54.598318 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.598261    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-4w4z2"
May 25 13:20:54.602437 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.602415    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
May 25 13:20:54.602714 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.602701    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-967tg\""
May 25 13:20:54.607227 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.607192    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-4w4z2"]
May 25 13:20:54.653788 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.653739    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2z2h\" (UniqueName: \"kubernetes.io/projected/659ee578-5137-48d8-8315-f9d40c932695-kube-api-access-s2z2h\") pod \"s3-tls-init-custom-4w4z2\" (UID: \"659ee578-5137-48d8-8315-f9d40c932695\") " pod="kserve/s3-tls-init-custom-4w4z2"
May 25 13:20:54.755102 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.755040    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s2z2h\" (UniqueName: \"kubernetes.io/projected/659ee578-5137-48d8-8315-f9d40c932695-kube-api-access-s2z2h\") pod \"s3-tls-init-custom-4w4z2\" (UID: \"659ee578-5137-48d8-8315-f9d40c932695\") " pod="kserve/s3-tls-init-custom-4w4z2"
May 25 13:20:54.764373 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.764347    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2z2h\" (UniqueName: \"kubernetes.io/projected/659ee578-5137-48d8-8315-f9d40c932695-kube-api-access-s2z2h\") pod \"s3-tls-init-custom-4w4z2\" (UID: \"659ee578-5137-48d8-8315-f9d40c932695\") " pod="kserve/s3-tls-init-custom-4w4z2"
May 25 13:20:54.908153 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:54.908119    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-4w4z2"
May 25 13:20:55.042200 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:55.042166    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-4w4z2"]
May 25 13:20:55.045571 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:20:55.045537    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod659ee578_5137_48d8_8315_f9d40c932695.slice/crio-697db2fb426ad043cfb1a149569077b353376ac438bf20e85939fa92abcfe458 WatchSource:0}: Error finding container 697db2fb426ad043cfb1a149569077b353376ac438bf20e85939fa92abcfe458: Status 404 returned error can't find the container with id 697db2fb426ad043cfb1a149569077b353376ac438bf20e85939fa92abcfe458
May 25 13:20:55.152232 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:20:55.152194    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-4w4z2" event={"ID":"659ee578-5137-48d8-8315-f9d40c932695","Type":"ContainerStarted","Data":"697db2fb426ad043cfb1a149569077b353376ac438bf20e85939fa92abcfe458"}
May 25 13:21:00.169286 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:00.169224    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-4w4z2" event={"ID":"659ee578-5137-48d8-8315-f9d40c932695","Type":"ContainerStarted","Data":"6b388c40eaa9966821ec170d694e8f4c6400980de15c8777411b887f07119825"}
May 25 13:21:00.193709 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:00.193650    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-4w4z2" podStartSLOduration=1.679412556 podStartE2EDuration="6.193634005s" podCreationTimestamp="2026-05-25 13:20:54 +0000 UTC" firstStartedPulling="2026-05-25 13:20:55.047443265 +0000 UTC m=+1005.086200367" lastFinishedPulling="2026-05-25 13:20:59.561664709 +0000 UTC m=+1009.600421816" observedRunningTime="2026-05-25 13:21:00.192981964 +0000 UTC m=+1010.231739098" watchObservedRunningTime="2026-05-25 13:21:00.193634005 +0000 UTC m=+1010.232391177"
May 25 13:21:05.182323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:05.182263    2549 generic.go:358] "Generic (PLEG): container finished" podID="659ee578-5137-48d8-8315-f9d40c932695" containerID="6b388c40eaa9966821ec170d694e8f4c6400980de15c8777411b887f07119825" exitCode=0
May 25 13:21:05.182323 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:05.182299    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-4w4z2" event={"ID":"659ee578-5137-48d8-8315-f9d40c932695","Type":"ContainerDied","Data":"6b388c40eaa9966821ec170d694e8f4c6400980de15c8777411b887f07119825"}
May 25 13:21:06.311427 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:06.311403    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-4w4z2"
May 25 13:21:06.353458 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:06.353418    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2z2h\" (UniqueName: \"kubernetes.io/projected/659ee578-5137-48d8-8315-f9d40c932695-kube-api-access-s2z2h\") pod \"659ee578-5137-48d8-8315-f9d40c932695\" (UID: \"659ee578-5137-48d8-8315-f9d40c932695\") "
May 25 13:21:06.355639 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:06.355609    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659ee578-5137-48d8-8315-f9d40c932695-kube-api-access-s2z2h" (OuterVolumeSpecName: "kube-api-access-s2z2h") pod "659ee578-5137-48d8-8315-f9d40c932695" (UID: "659ee578-5137-48d8-8315-f9d40c932695"). InnerVolumeSpecName "kube-api-access-s2z2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:21:06.454556 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:06.454457    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2z2h\" (UniqueName: \"kubernetes.io/projected/659ee578-5137-48d8-8315-f9d40c932695-kube-api-access-s2z2h\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:21:07.188664 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.188628    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-4w4z2" event={"ID":"659ee578-5137-48d8-8315-f9d40c932695","Type":"ContainerDied","Data":"697db2fb426ad043cfb1a149569077b353376ac438bf20e85939fa92abcfe458"}
May 25 13:21:07.188664 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.188669    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697db2fb426ad043cfb1a149569077b353376ac438bf20e85939fa92abcfe458"
May 25 13:21:07.188920 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.188634    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-4w4z2"
May 25 13:21:07.801531 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.801485    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"]
May 25 13:21:07.801931 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.801776    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659ee578-5137-48d8-8315-f9d40c932695" containerName="s3-tls-init-custom"
May 25 13:21:07.801931 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.801789    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="659ee578-5137-48d8-8315-f9d40c932695" containerName="s3-tls-init-custom"
May 25 13:21:07.801931 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.801834    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="659ee578-5137-48d8-8315-f9d40c932695" containerName="s3-tls-init-custom"
May 25 13:21:07.804731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.804712    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:07.807146 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.807126    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
May 25 13:21:07.807289 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.807159    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
May 25 13:21:07.807413 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.807400    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-967tg\""
May 25 13:21:07.812644 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.812617    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"]
May 25 13:21:07.864985 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.864937    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:07.864985 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.864983    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07690b93-a4a3-4b9e-94ff-5ae135d57aef-data\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:07.865216 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.865066    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlwk\" (UniqueName: \"kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-kube-api-access-bmlwk\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:07.966058 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.966015    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07690b93-a4a3-4b9e-94ff-5ae135d57aef-data\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:07.966226 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.966086    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlwk\" (UniqueName: \"kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-kube-api-access-bmlwk\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:07.966226 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.966149    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:07.966390 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:21:07.966258    2549 projected.go:264] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
May 25 13:21:07.966390 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:21:07.966298    2549 projected.go:194] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-7fd5766db9-2dt72: secret "seaweedfs-tls-serving" not found
May 25 13:21:07.969626 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:21:07.966744    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-seaweedfs-tls-serving podName:07690b93-a4a3-4b9e-94ff-5ae135d57aef nodeName:}" failed. No retries permitted until 2026-05-25 13:21:08.466697928 +0000 UTC m=+1018.505455029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-7fd5766db9-2dt72" (UID: "07690b93-a4a3-4b9e-94ff-5ae135d57aef") : secret "seaweedfs-tls-serving" not found
May 25 13:21:07.969626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.966832    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07690b93-a4a3-4b9e-94ff-5ae135d57aef-data\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:07.976716 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:07.976686    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlwk\" (UniqueName: \"kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-kube-api-access-bmlwk\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:08.471092 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:08.471039    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:08.473673 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:08.473637    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/07690b93-a4a3-4b9e-94ff-5ae135d57aef-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-2dt72\" (UID: \"07690b93-a4a3-4b9e-94ff-5ae135d57aef\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:08.714553 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:08.714488    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"
May 25 13:21:08.853747 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:08.853700    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-2dt72"]
May 25 13:21:08.857144 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:21:08.857091    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07690b93_a4a3_4b9e_94ff_5ae135d57aef.slice/crio-74b2ee0b028c6941e266b9f2011938075faf8cbcc0a074a232b18827f647325e WatchSource:0}: Error finding container 74b2ee0b028c6941e266b9f2011938075faf8cbcc0a074a232b18827f647325e: Status 404 returned error can't find the container with id 74b2ee0b028c6941e266b9f2011938075faf8cbcc0a074a232b18827f647325e
May 25 13:21:09.195099 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:09.195057    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72" event={"ID":"07690b93-a4a3-4b9e-94ff-5ae135d57aef","Type":"ContainerStarted","Data":"74b2ee0b028c6941e266b9f2011938075faf8cbcc0a074a232b18827f647325e"}
May 25 13:21:11.668540 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:11.668508    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
May 25 13:21:12.205945 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:12.205902    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72" event={"ID":"07690b93-a4a3-4b9e-94ff-5ae135d57aef","Type":"ContainerStarted","Data":"f87257689245e488bd5c93ae1f6a55656c20525ba8a5877176d74140d35cceff"}
May 25 13:21:12.225433 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:12.225378    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-7fd5766db9-2dt72" podStartSLOduration=2.422974289 podStartE2EDuration="5.225361093s" podCreationTimestamp="2026-05-25 13:21:07 +0000 UTC" firstStartedPulling="2026-05-25 13:21:08.858464398 +0000 UTC m=+1018.897221501" lastFinishedPulling="2026-05-25 13:21:11.660851202 +0000 UTC m=+1021.699608305" observedRunningTime="2026-05-25 13:21:12.224019746 +0000 UTC m=+1022.262776871" watchObservedRunningTime="2026-05-25 13:21:12.225361093 +0000 UTC m=+1022.264118217"
May 25 13:21:33.451437 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.451398    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"]
May 25 13:21:33.485148 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.485114    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"]
May 25 13:21:33.485325 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.485282    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.487786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.487763    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:21:33.488042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.488026    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
May 25 13:21:33.488120 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.488070    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:21:33.488177 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.488141    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zwsdr\""
May 25 13:21:33.488177 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.488149    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
May 25 13:21:33.551957 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.551922    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.552138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.551992    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.552138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.552024    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.552138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.552043    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6cxr\" (UniqueName: \"kubernetes.io/projected/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kube-api-access-n6cxr\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.652797 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.652755    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.652974 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.652808    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.652974 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.652936    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n6cxr\" (UniqueName: \"kubernetes.io/projected/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kube-api-access-n6cxr\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.653071 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.653002    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.653358 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.653340    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.653539 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.653519    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.655388 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.655370    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.661232 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.661210    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6cxr\" (UniqueName: \"kubernetes.io/projected/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kube-api-access-n6cxr\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-8ggcn\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.795432 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.795333    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:33.929158 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:33.929123    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"]
May 25 13:21:33.932157 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:21:33.932121    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede8d9fc_f706_4bee_8e3e_cd29dc5bb406.slice/crio-79ebf3d931eed2be54b19cd79ce6d548074b8b8078ece70dccf7d24a1d6917d4 WatchSource:0}: Error finding container 79ebf3d931eed2be54b19cd79ce6d548074b8b8078ece70dccf7d24a1d6917d4: Status 404 returned error can't find the container with id 79ebf3d931eed2be54b19cd79ce6d548074b8b8078ece70dccf7d24a1d6917d4
May 25 13:21:34.264725 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:34.264691    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerStarted","Data":"79ebf3d931eed2be54b19cd79ce6d548074b8b8078ece70dccf7d24a1d6917d4"}
May 25 13:21:37.274897 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:37.274809    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerStarted","Data":"b7a60fe37de6281f85310a6d0675abb5035b96e71a5044827c57244785a2f230"}
May 25 13:21:41.286153 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:41.286108    2549 generic.go:358] "Generic (PLEG): container finished" podID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerID="b7a60fe37de6281f85310a6d0675abb5035b96e71a5044827c57244785a2f230" exitCode=0
May 25 13:21:41.286153 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:41.286152    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerDied","Data":"b7a60fe37de6281f85310a6d0675abb5035b96e71a5044827c57244785a2f230"}
May 25 13:21:54.329937 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:54.329898    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerStarted","Data":"43c67d0a04b7026c8007784bfc4b428ac62754d4d5aa43f2729c72e9e70bbd32"}
May 25 13:21:56.337340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:56.337229    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerStarted","Data":"ba602eab501864169ee5932348222d01afcada4b7bd1a9aec7f1aa64a4b69b66"}
May 25 13:21:59.348571 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:59.348532    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerStarted","Data":"ac2d55043713b002d850ebd6739213495cb9693a64b2e8d62ef81ed9baeff3cc"}
May 25 13:21:59.349017 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:59.348788    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:59.349017 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:59.348822    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:59.349017 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:59.348835    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:21:59.350146 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:59.350115    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:21:59.350727 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:59.350707    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:21:59.396944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:21:59.396893    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podStartSLOduration=1.705032576 podStartE2EDuration="26.396878848s" podCreationTimestamp="2026-05-25 13:21:33 +0000 UTC" firstStartedPulling="2026-05-25 13:21:33.933898973 +0000 UTC m=+1043.972656076" lastFinishedPulling="2026-05-25 13:21:58.625745247 +0000 UTC m=+1068.664502348" observedRunningTime="2026-05-25 13:21:59.39428905 +0000 UTC m=+1069.433046168" watchObservedRunningTime="2026-05-25 13:21:59.396878848 +0000 UTC m=+1069.435635973"
May 25 13:22:00.351573 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:00.351535    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:22:00.352018 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:00.352000    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:22:00.355320 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:00.355300    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:22:01.354042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:01.353997    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:22:01.354508 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:01.354312    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:22:11.354033 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:11.353990    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:22:11.354521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:11.354456    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:22:21.354486 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:21.354357    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:22:21.622532 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:21.354721    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:22:31.354359 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:31.354312    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:22:31.354820 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:31.354797    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:22:41.354503 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:41.354448    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:22:41.355008 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:41.354984    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:22:51.354210 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:51.354154    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:22:51.354717 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:22:51.354597    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:01.354971 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:01.354941    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:23:01.355401 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:01.355224    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:23:08.564994 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.564956    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"]
May 25 13:23:08.565399 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.565370    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" containerID="cri-o://43c67d0a04b7026c8007784bfc4b428ac62754d4d5aa43f2729c72e9e70bbd32" gracePeriod=30
May 25 13:23:08.565455 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.565409    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" containerID="cri-o://ac2d55043713b002d850ebd6739213495cb9693a64b2e8d62ef81ed9baeff3cc" gracePeriod=30
May 25 13:23:08.565500 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.565434    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy" containerID="cri-o://ba602eab501864169ee5932348222d01afcada4b7bd1a9aec7f1aa64a4b69b66" gracePeriod=30
May 25 13:23:08.750477 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.750440    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"]
May 25 13:23:08.753614 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.753597    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.756030 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.756008    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
May 25 13:23:08.756134 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.756045    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
May 25 13:23:08.770914 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.770884    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"]
May 25 13:23:08.856922 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.856826    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.856922 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.856873    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfzv\" (UniqueName: \"kubernetes.io/projected/628ce220-e587-4394-a171-9c49fb761b87-kube-api-access-ggfzv\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.856922 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.856894    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/628ce220-e587-4394-a171-9c49fb761b87-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.857184 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.857005    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/628ce220-e587-4394-a171-9c49fb761b87-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.958121 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.958070    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.958121 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.958130    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfzv\" (UniqueName: \"kubernetes.io/projected/628ce220-e587-4394-a171-9c49fb761b87-kube-api-access-ggfzv\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.958433 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.958153    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/628ce220-e587-4394-a171-9c49fb761b87-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.958433 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.958187    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/628ce220-e587-4394-a171-9c49fb761b87-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.958433 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:23:08.958249    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
May 25 13:23:08.958433 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:23:08.958354    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls podName:628ce220-e587-4394-a171-9c49fb761b87 nodeName:}" failed. No retries permitted until 2026-05-25 13:23:09.458332323 +0000 UTC m=+1139.497089439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" (UID: "628ce220-e587-4394-a171-9c49fb761b87") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
May 25 13:23:08.958638 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.958590    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/628ce220-e587-4394-a171-9c49fb761b87-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.958904 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.958886    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/628ce220-e587-4394-a171-9c49fb761b87-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:08.980566 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:08.980541    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfzv\" (UniqueName: \"kubernetes.io/projected/628ce220-e587-4394-a171-9c49fb761b87-kube-api-access-ggfzv\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:09.462619 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:09.462575    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:09.464993 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:09.464970    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:09.550467 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:09.550433    2549 generic.go:358] "Generic (PLEG): container finished" podID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerID="ba602eab501864169ee5932348222d01afcada4b7bd1a9aec7f1aa64a4b69b66" exitCode=2
May 25 13:23:09.550467 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:09.550470    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerDied","Data":"ba602eab501864169ee5932348222d01afcada4b7bd1a9aec7f1aa64a4b69b66"}
May 25 13:23:09.663200 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:09.663148    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:09.801423 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:09.801385    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"]
May 25 13:23:09.804871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:23:09.804838    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628ce220_e587_4394_a171_9c49fb761b87.slice/crio-4fdf35a117694c9d06309780e38c88fb694c8d7573113bb56f287c6d35eb26f3 WatchSource:0}: Error finding container 4fdf35a117694c9d06309780e38c88fb694c8d7573113bb56f287c6d35eb26f3: Status 404 returned error can't find the container with id 4fdf35a117694c9d06309780e38c88fb694c8d7573113bb56f287c6d35eb26f3
May 25 13:23:09.806738 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:09.806721    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:23:10.352291 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:10.352228    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 25 13:23:10.555577 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:10.555536    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerStarted","Data":"0abc0291495bf0996176d4992814d6ccd7f698d62ceed7924c0740828e927810"}
May 25 13:23:10.555577 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:10.555579    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerStarted","Data":"4fdf35a117694c9d06309780e38c88fb694c8d7573113bb56f287c6d35eb26f3"}
May 25 13:23:11.354205 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:11.354167    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:23:11.354613 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:11.354491    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:13.564218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:13.564188    2549 generic.go:358] "Generic (PLEG): container finished" podID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerID="43c67d0a04b7026c8007784bfc4b428ac62754d4d5aa43f2729c72e9e70bbd32" exitCode=0
May 25 13:23:13.564609 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:13.564252    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerDied","Data":"43c67d0a04b7026c8007784bfc4b428ac62754d4d5aa43f2729c72e9e70bbd32"}
May 25 13:23:13.565587 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:13.565564    2549 generic.go:358] "Generic (PLEG): container finished" podID="628ce220-e587-4394-a171-9c49fb761b87" containerID="0abc0291495bf0996176d4992814d6ccd7f698d62ceed7924c0740828e927810" exitCode=0
May 25 13:23:13.565696 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:13.565595    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerDied","Data":"0abc0291495bf0996176d4992814d6ccd7f698d62ceed7924c0740828e927810"}
May 25 13:23:14.570924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:14.570883    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerStarted","Data":"7abdb5050a98c16e4b084e6e33e216114ff88b4bf6b814e9c31a6fc014e6de46"}
May 25 13:23:14.571347 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:14.570931    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerStarted","Data":"6f73a440d06874b8bcef5ebc176b259dc97b166a3267525b87ed0afce7529e0f"}
May 25 13:23:14.571347 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:14.570944    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerStarted","Data":"700933ac8730c347aa981b62e367063df6b7b665cd7a79b51e7e0d41fdf01805"}
May 25 13:23:14.571347 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:14.571247    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:14.571347 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:14.571302    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:14.572662 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:14.572634    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:23:14.594301 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:14.594243    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podStartSLOduration=6.594230033 podStartE2EDuration="6.594230033s" podCreationTimestamp="2026-05-25 13:23:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:23:14.592507464 +0000 UTC m=+1144.631264588" watchObservedRunningTime="2026-05-25 13:23:14.594230033 +0000 UTC m=+1144.632987157"
May 25 13:23:15.352788 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:15.352744    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 25 13:23:15.573674 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:15.573637    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:15.574088 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:15.573732    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:23:15.574526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:15.574504    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:16.577234 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:16.577192    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:23:16.577688 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:16.577660    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:20.352526 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:20.352437    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 25 13:23:20.352920 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:20.352565    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:23:21.354600 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:21.354551    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:23:21.354955 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:21.354832    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:21.581022 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:21.580938    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:23:21.581563 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:21.581524    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:23:21.581855 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:21.581834    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:25.351732 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:25.351686    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 25 13:23:30.352050 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:30.352003    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 25 13:23:31.354613 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:31.354570    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 25 13:23:31.355057 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:31.354725    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:23:31.355057 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:31.354891    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:31.355057 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:31.354974    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:23:31.582244 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:31.582199    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:23:31.582639 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:31.582611    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:35.352786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:35.352744    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 25 13:23:38.638257 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.638223    2549 generic.go:358] "Generic (PLEG): container finished" podID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerID="ac2d55043713b002d850ebd6739213495cb9693a64b2e8d62ef81ed9baeff3cc" exitCode=0
May 25 13:23:38.638631 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.638306    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerDied","Data":"ac2d55043713b002d850ebd6739213495cb9693a64b2e8d62ef81ed9baeff3cc"}
May 25 13:23:38.718122 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.718096    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:23:38.801327 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.801292    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-proxy-tls\") pod \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") "
May 25 13:23:38.801499 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.801362    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6cxr\" (UniqueName: \"kubernetes.io/projected/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kube-api-access-n6cxr\") pod \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") "
May 25 13:23:38.801499 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.801422    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kserve-provision-location\") pod \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") "
May 25 13:23:38.801789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.801754    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" (UID: "ede8d9fc-f706-4bee-8e3e-cd29dc5bb406"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:23:38.803546 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.803516    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" (UID: "ede8d9fc-f706-4bee-8e3e-cd29dc5bb406"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:23:38.803627 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.803548    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kube-api-access-n6cxr" (OuterVolumeSpecName: "kube-api-access-n6cxr") pod "ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" (UID: "ede8d9fc-f706-4bee-8e3e-cd29dc5bb406"). InnerVolumeSpecName "kube-api-access-n6cxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:23:38.902652 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.902616    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\" (UID: \"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406\") "
May 25 13:23:38.902806 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.902766    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:23:38.902806 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.902787    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6cxr\" (UniqueName: \"kubernetes.io/projected/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kube-api-access-n6cxr\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:23:38.902806 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.902803    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:23:38.902973 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:38.902955    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" (UID: "ede8d9fc-f706-4bee-8e3e-cd29dc5bb406"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:23:39.003868 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.003832    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:23:39.642634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.642599    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn" event={"ID":"ede8d9fc-f706-4bee-8e3e-cd29dc5bb406","Type":"ContainerDied","Data":"79ebf3d931eed2be54b19cd79ce6d548074b8b8078ece70dccf7d24a1d6917d4"}
May 25 13:23:39.642634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.642635    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"
May 25 13:23:39.643049 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.642651    2549 scope.go:117] "RemoveContainer" containerID="ac2d55043713b002d850ebd6739213495cb9693a64b2e8d62ef81ed9baeff3cc"
May 25 13:23:39.650121 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.650100    2549 scope.go:117] "RemoveContainer" containerID="ba602eab501864169ee5932348222d01afcada4b7bd1a9aec7f1aa64a4b69b66"
May 25 13:23:39.657239 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.657222    2549 scope.go:117] "RemoveContainer" containerID="43c67d0a04b7026c8007784bfc4b428ac62754d4d5aa43f2729c72e9e70bbd32"
May 25 13:23:39.664110 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.664095    2549 scope.go:117] "RemoveContainer" containerID="b7a60fe37de6281f85310a6d0675abb5035b96e71a5044827c57244785a2f230"
May 25 13:23:39.664533 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.664513    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"]
May 25 13:23:39.668835 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:39.668814    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-8ggcn"]
May 25 13:23:40.497257 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:40.497220    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" path="/var/lib/kubelet/pods/ede8d9fc-f706-4bee-8e3e-cd29dc5bb406/volumes"
May 25 13:23:41.582244 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:41.582194    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:23:41.582684 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:41.582655    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:23:51.581511 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:51.581465    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:23:51.582005 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:23:51.581880    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:24:01.581636 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:01.581592    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:24:01.582117 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:01.582046    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:24:11.060872 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:11.060842    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:24:11.062027 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:11.062009    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:24:11.581518 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:11.581476    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:24:11.581891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:11.581869    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:24:21.582210 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:21.582120    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:24:21.582638 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:21.582248    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:24:33.777109 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.777077    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"]
May 25 13:24:33.777641 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.777440    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" containerID="cri-o://700933ac8730c347aa981b62e367063df6b7b665cd7a79b51e7e0d41fdf01805" gracePeriod=30
May 25 13:24:33.777641 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.777483    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" containerID="cri-o://7abdb5050a98c16e4b084e6e33e216114ff88b4bf6b814e9c31a6fc014e6de46" gracePeriod=30
May 25 13:24:33.777641 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.777483    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy" containerID="cri-o://6f73a440d06874b8bcef5ebc176b259dc97b166a3267525b87ed0afce7529e0f" gracePeriod=30
May 25 13:24:33.858379 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858342    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"]
May 25 13:24:33.858760 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858744    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent"
May 25 13:24:33.858810 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858765    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent"
May 25 13:24:33.858810 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858778    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="storage-initializer"
May 25 13:24:33.858810 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858786    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="storage-initializer"
May 25 13:24:33.858810 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858797    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy"
May 25 13:24:33.858810 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858806    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy"
May 25 13:24:33.858952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858823    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container"
May 25 13:24:33.858952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858828    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container"
May 25 13:24:33.858952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858877    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="agent"
May 25 13:24:33.858952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858886    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kserve-container"
May 25 13:24:33.858952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.858894    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ede8d9fc-f706-4bee-8e3e-cd29dc5bb406" containerName="kube-rbac-proxy"
May 25 13:24:33.861844 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.861828    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:33.864118 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.864099    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
May 25 13:24:33.864214 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.864131    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
May 25 13:24:33.870615 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.870595    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"]
May 25 13:24:33.889968 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.889945    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d598bc-f1f6-4998-993d-e750b6600165-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:33.890076 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.889999    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:33.890076 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.890046    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7mn\" (UniqueName: \"kubernetes.io/projected/66d598bc-f1f6-4998-993d-e750b6600165-kube-api-access-rb7mn\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:33.991027 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.990982    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d598bc-f1f6-4998-993d-e750b6600165-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:33.991197 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.991059    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:33.991197 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.991111    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7mn\" (UniqueName: \"kubernetes.io/projected/66d598bc-f1f6-4998-993d-e750b6600165-kube-api-access-rb7mn\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:33.991335 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:24:33.991215    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
May 25 13:24:33.991335 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:24:33.991296    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls podName:66d598bc-f1f6-4998-993d-e750b6600165 nodeName:}" failed. No retries permitted until 2026-05-25 13:24:34.4912596 +0000 UTC m=+1224.530016707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls") pod "message-dumper-predictor-c7d86bcbd-ncqv8" (UID: "66d598bc-f1f6-4998-993d-e750b6600165") : secret "message-dumper-predictor-serving-cert" not found
May 25 13:24:33.991627 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:33.991605    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d598bc-f1f6-4998-993d-e750b6600165-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:34.002260 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:34.002239    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7mn\" (UniqueName: \"kubernetes.io/projected/66d598bc-f1f6-4998-993d-e750b6600165-kube-api-access-rb7mn\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:34.494097 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:34.494056    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:34.496712 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:34.496677    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-ncqv8\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:34.772313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:34.772193    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:34.798382 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:34.798347    2549 generic.go:358] "Generic (PLEG): container finished" podID="628ce220-e587-4394-a171-9c49fb761b87" containerID="6f73a440d06874b8bcef5ebc176b259dc97b166a3267525b87ed0afce7529e0f" exitCode=2
May 25 13:24:34.798825 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:34.798417    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerDied","Data":"6f73a440d06874b8bcef5ebc176b259dc97b166a3267525b87ed0afce7529e0f"}
May 25 13:24:34.897410 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:34.897290    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"]
May 25 13:24:34.899968 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:24:34.899940    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d598bc_f1f6_4998_993d_e750b6600165.slice/crio-30111adc5082aa8048a01c01194e3679448a70a4a64913e768b3d86037be6c83 WatchSource:0}: Error finding container 30111adc5082aa8048a01c01194e3679448a70a4a64913e768b3d86037be6c83: Status 404 returned error can't find the container with id 30111adc5082aa8048a01c01194e3679448a70a4a64913e768b3d86037be6c83
May 25 13:24:35.802917 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:35.802876    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" event={"ID":"66d598bc-f1f6-4998-993d-e750b6600165","Type":"ContainerStarted","Data":"30111adc5082aa8048a01c01194e3679448a70a4a64913e768b3d86037be6c83"}
May 25 13:24:36.578352 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:36.578263    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
May 25 13:24:36.807171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:36.807132    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" event={"ID":"66d598bc-f1f6-4998-993d-e750b6600165","Type":"ContainerStarted","Data":"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27"}
May 25 13:24:36.807171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:36.807167    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" event={"ID":"66d598bc-f1f6-4998-993d-e750b6600165","Type":"ContainerStarted","Data":"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b"}
May 25 13:24:36.807708 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:36.807259    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:36.824641 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:36.824595    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" podStartSLOduration=2.698356695 podStartE2EDuration="3.824579851s" podCreationTimestamp="2026-05-25 13:24:33 +0000 UTC" firstStartedPulling="2026-05-25 13:24:34.901794042 +0000 UTC m=+1224.940551147" lastFinishedPulling="2026-05-25 13:24:36.028017201 +0000 UTC m=+1226.066774303" observedRunningTime="2026-05-25 13:24:36.822990416 +0000 UTC m=+1226.861747550" watchObservedRunningTime="2026-05-25 13:24:36.824579851 +0000 UTC m=+1226.863336969"
May 25 13:24:37.809814 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:37.809778    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:37.811814 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:37.811788    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:38.813773 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:38.813737    2549 generic.go:358] "Generic (PLEG): container finished" podID="628ce220-e587-4394-a171-9c49fb761b87" containerID="700933ac8730c347aa981b62e367063df6b7b665cd7a79b51e7e0d41fdf01805" exitCode=0
May 25 13:24:38.814139 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:38.813810    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerDied","Data":"700933ac8730c347aa981b62e367063df6b7b665cd7a79b51e7e0d41fdf01805"}
May 25 13:24:41.577842 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:41.577797    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
May 25 13:24:41.582088 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:41.582054    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:24:41.582490 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:41.582465    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:24:44.821159 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:44.821126    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:24:46.577719 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:46.577668    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
May 25 13:24:46.578117 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:46.577795    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:24:51.578533 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:51.578441    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
May 25 13:24:51.581798 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:51.581774    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:24:51.582079 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:51.582062    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:24:53.930674 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:53.930635    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"]
May 25 13:24:53.934253 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:53.934237    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:53.936431 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:53.936409    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
May 25 13:24:53.936537 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:53.936458    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
May 25 13:24:53.947556 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:53.947529    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"]
May 25 13:24:54.046256 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.046215    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/555a3549-22bc-4739-811b-38f36ea8868a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.046442 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.046317    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/555a3549-22bc-4739-811b-38f36ea8868a-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.046442 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.046356    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8nc\" (UniqueName: \"kubernetes.io/projected/555a3549-22bc-4739-811b-38f36ea8868a-kube-api-access-cw8nc\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.046442 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.046412    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.147715 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.147667    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/555a3549-22bc-4739-811b-38f36ea8868a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.147905 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.147730    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/555a3549-22bc-4739-811b-38f36ea8868a-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.147905 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.147757    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8nc\" (UniqueName: \"kubernetes.io/projected/555a3549-22bc-4739-811b-38f36ea8868a-kube-api-access-cw8nc\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.147905 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.147837    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.148056 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:24:54.147949    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
May 25 13:24:54.148056 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:24:54.148002    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls podName:555a3549-22bc-4739-811b-38f36ea8868a nodeName:}" failed. No retries permitted until 2026-05-25 13:24:54.647988428 +0000 UTC m=+1244.686745529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls") pod "isvc-logger-predictor-6d94fc7bdd-7xvh7" (UID: "555a3549-22bc-4739-811b-38f36ea8868a") : secret "isvc-logger-predictor-serving-cert" not found
May 25 13:24:54.148231 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.148213    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/555a3549-22bc-4739-811b-38f36ea8868a-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.148405 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.148387    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/555a3549-22bc-4739-811b-38f36ea8868a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.160964 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.160939    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8nc\" (UniqueName: \"kubernetes.io/projected/555a3549-22bc-4739-811b-38f36ea8868a-kube-api-access-cw8nc\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.653902 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.653864    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.656385 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.656360    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-7xvh7\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.843608 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.843566    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:54.976172 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:54.976051    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"]
May 25 13:24:54.978806 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:24:54.978776    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555a3549_22bc_4739_811b_38f36ea8868a.slice/crio-54e7d672ffa6ab3d7a85bb868e717da1d3e01aed5807a8849768b4d903ca2f4a WatchSource:0}: Error finding container 54e7d672ffa6ab3d7a85bb868e717da1d3e01aed5807a8849768b4d903ca2f4a: Status 404 returned error can't find the container with id 54e7d672ffa6ab3d7a85bb868e717da1d3e01aed5807a8849768b4d903ca2f4a
May 25 13:24:55.861888 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:55.861845    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerStarted","Data":"e9a5eaefb27b6963893b9ee2bd62a94c39c7f2a37a1b5f710236f3937b0d5cd6"}
May 25 13:24:55.861888 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:55.861891    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerStarted","Data":"54e7d672ffa6ab3d7a85bb868e717da1d3e01aed5807a8849768b4d903ca2f4a"}
May 25 13:24:56.578061 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:56.578020    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
May 25 13:24:58.871889 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:58.871855    2549 generic.go:358] "Generic (PLEG): container finished" podID="555a3549-22bc-4739-811b-38f36ea8868a" containerID="e9a5eaefb27b6963893b9ee2bd62a94c39c7f2a37a1b5f710236f3937b0d5cd6" exitCode=0
May 25 13:24:58.872306 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:58.871900    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerDied","Data":"e9a5eaefb27b6963893b9ee2bd62a94c39c7f2a37a1b5f710236f3937b0d5cd6"}
May 25 13:24:59.876963 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:59.876926    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerStarted","Data":"dc0f09ec85270489a8de9a901a046bb50315b180ebb894f7bf980576aa146325"}
May 25 13:24:59.876963 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:59.876967    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerStarted","Data":"f393fbf07816a1b35ee5c322b8f6e4b7a55d6031675a16864d37c1c3b46ae9bb"}
May 25 13:24:59.877430 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:59.876981    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerStarted","Data":"fcf178c67ad7df368fef4a63f0e0fbe5421903ea35786d3b77a1e6cd542294d0"}
May 25 13:24:59.877430 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:59.877289    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:59.877515 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:59.877437    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:24:59.878685 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:59.878658    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:24:59.896338 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:24:59.896298    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podStartSLOduration=6.896286065 podStartE2EDuration="6.896286065s" podCreationTimestamp="2026-05-25 13:24:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:24:59.895344507 +0000 UTC m=+1249.934101631" watchObservedRunningTime="2026-05-25 13:24:59.896286065 +0000 UTC m=+1249.935043195"
May 25 13:25:00.880354 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:00.880325    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:25:00.880750 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:00.880496    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:25:00.881559 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:00.881533    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:01.578325 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:01.578243    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
May 25 13:25:01.581512 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:01.581482    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
May 25 13:25:01.581644 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:01.581626    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:25:01.581935 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:01.581912    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:01.582026 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:01.582013    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:25:01.883663 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:01.883623    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:25:01.884055 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:01.884008    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:03.891826 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:03.891786    2549 generic.go:358] "Generic (PLEG): container finished" podID="628ce220-e587-4394-a171-9c49fb761b87" containerID="7abdb5050a98c16e4b084e6e33e216114ff88b4bf6b814e9c31a6fc014e6de46" exitCode=0
May 25 13:25:03.892287 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:03.891831    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerDied","Data":"7abdb5050a98c16e4b084e6e33e216114ff88b4bf6b814e9c31a6fc014e6de46"}
May 25 13:25:03.929634 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:03.929610    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:25:04.027832 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.027796    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls\") pod \"628ce220-e587-4394-a171-9c49fb761b87\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") "
May 25 13:25:04.027992 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.027858    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggfzv\" (UniqueName: \"kubernetes.io/projected/628ce220-e587-4394-a171-9c49fb761b87-kube-api-access-ggfzv\") pod \"628ce220-e587-4394-a171-9c49fb761b87\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") "
May 25 13:25:04.027992 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.027883    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/628ce220-e587-4394-a171-9c49fb761b87-kserve-provision-location\") pod \"628ce220-e587-4394-a171-9c49fb761b87\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") "
May 25 13:25:04.027992 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.027931    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/628ce220-e587-4394-a171-9c49fb761b87-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"628ce220-e587-4394-a171-9c49fb761b87\" (UID: \"628ce220-e587-4394-a171-9c49fb761b87\") "
May 25 13:25:04.028238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.028216    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628ce220-e587-4394-a171-9c49fb761b87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "628ce220-e587-4394-a171-9c49fb761b87" (UID: "628ce220-e587-4394-a171-9c49fb761b87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:25:04.028354 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.028327    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628ce220-e587-4394-a171-9c49fb761b87-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "628ce220-e587-4394-a171-9c49fb761b87" (UID: "628ce220-e587-4394-a171-9c49fb761b87"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:25:04.030122 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.030098    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "628ce220-e587-4394-a171-9c49fb761b87" (UID: "628ce220-e587-4394-a171-9c49fb761b87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:25:04.030196 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.030149    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628ce220-e587-4394-a171-9c49fb761b87-kube-api-access-ggfzv" (OuterVolumeSpecName: "kube-api-access-ggfzv") pod "628ce220-e587-4394-a171-9c49fb761b87" (UID: "628ce220-e587-4394-a171-9c49fb761b87"). InnerVolumeSpecName "kube-api-access-ggfzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:25:04.128473 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.128429    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/628ce220-e587-4394-a171-9c49fb761b87-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:25:04.128473 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.128465    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/628ce220-e587-4394-a171-9c49fb761b87-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:25:04.128473 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.128476    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggfzv\" (UniqueName: \"kubernetes.io/projected/628ce220-e587-4394-a171-9c49fb761b87-kube-api-access-ggfzv\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:25:04.128712 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.128496    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/628ce220-e587-4394-a171-9c49fb761b87-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:25:04.903682 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.903644    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r" event={"ID":"628ce220-e587-4394-a171-9c49fb761b87","Type":"ContainerDied","Data":"4fdf35a117694c9d06309780e38c88fb694c8d7573113bb56f287c6d35eb26f3"}
May 25 13:25:04.903682 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.903691    2549 scope.go:117] "RemoveContainer" containerID="7abdb5050a98c16e4b084e6e33e216114ff88b4bf6b814e9c31a6fc014e6de46"
May 25 13:25:04.904245 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.903688    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"
May 25 13:25:04.911448 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.911427    2549 scope.go:117] "RemoveContainer" containerID="6f73a440d06874b8bcef5ebc176b259dc97b166a3267525b87ed0afce7529e0f"
May 25 13:25:04.918624 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.918532    2549 scope.go:117] "RemoveContainer" containerID="700933ac8730c347aa981b62e367063df6b7b665cd7a79b51e7e0d41fdf01805"
May 25 13:25:04.920692 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.920669    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"]
May 25 13:25:04.923834 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.923811    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-vnj8r"]
May 25 13:25:04.928039 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:04.928022    2549 scope.go:117] "RemoveContainer" containerID="0abc0291495bf0996176d4992814d6ccd7f698d62ceed7924c0740828e927810"
May 25 13:25:06.497677 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:06.497641    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="628ce220-e587-4394-a171-9c49fb761b87" path="/var/lib/kubelet/pods/628ce220-e587-4394-a171-9c49fb761b87/volumes"
May 25 13:25:06.887633 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:06.887603    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:25:06.888171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:06.888134    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:25:06.888408 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:06.888376    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:16.888117 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:16.888064    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:25:16.888539 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:16.888502    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:26.888284 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:26.888218    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:25:26.888739 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:26.888704    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:36.888422 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:36.888366    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:25:36.888896 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:36.888824    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:46.888532 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:46.888481    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:25:46.888969 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:46.888944    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:25:56.888536 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:56.888494    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:25:56.888951 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:25:56.888928    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:06.888980 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:06.888950    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:26:06.889423 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:06.889180    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:26:18.933924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:18.933849    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-ncqv8_66d598bc-f1f6-4998-993d-e750b6600165/kserve-container/0.log"
May 25 13:26:19.134793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.134758    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"]
May 25 13:26:19.135156 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.135122    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" containerID="cri-o://fcf178c67ad7df368fef4a63f0e0fbe5421903ea35786d3b77a1e6cd542294d0" gracePeriod=30
May 25 13:26:19.135350 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.135157    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" containerID="cri-o://dc0f09ec85270489a8de9a901a046bb50315b180ebb894f7bf980576aa146325" gracePeriod=30
May 25 13:26:19.135350 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.135181    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy" containerID="cri-o://f393fbf07816a1b35ee5c322b8f6e4b7a55d6031675a16864d37c1c3b46ae9bb" gracePeriod=30
May 25 13:26:19.165832 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.165803    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"]
May 25 13:26:19.166073 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166061    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="storage-initializer"
May 25 13:26:19.166113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166074    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="storage-initializer"
May 25 13:26:19.166113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166085    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container"
May 25 13:26:19.166113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166091    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container"
May 25 13:26:19.166113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166098    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy"
May 25 13:26:19.166113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166104    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy"
May 25 13:26:19.166113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166109    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent"
May 25 13:26:19.166113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166114    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent"
May 25 13:26:19.166340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166167    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kube-rbac-proxy"
May 25 13:26:19.166340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166174    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="agent"
May 25 13:26:19.166340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.166183    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="628ce220-e587-4394-a171-9c49fb761b87" containerName="kserve-container"
May 25 13:26:19.169170 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.169155    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.173796 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.172291    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
May 25 13:26:19.173796 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.172599    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
May 25 13:26:19.181944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.181917    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"]
May 25 13:26:19.243000 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.242915    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"]
May 25 13:26:19.243214 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.243176    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" podUID="66d598bc-f1f6-4998-993d-e750b6600165" containerName="kserve-container" containerID="cri-o://269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b" gracePeriod=30
May 25 13:26:19.243372 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.243224    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" podUID="66d598bc-f1f6-4998-993d-e750b6600165" containerName="kube-rbac-proxy" containerID="cri-o://e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27" gracePeriod=30
May 25 13:26:19.313906 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.313873    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.314048 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.313916    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.314048 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.314007    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzth\" (UniqueName: \"kubernetes.io/projected/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kube-api-access-wqzth\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.314048 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.314026    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.414777 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.414743    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzth\" (UniqueName: \"kubernetes.io/projected/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kube-api-access-wqzth\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.414921 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.414801    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.414921 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.414832    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.414921 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.414860    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.415368 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.415345    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.415638 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.415615    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.417708 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.417682    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.423450 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.423418    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzth\" (UniqueName: \"kubernetes.io/projected/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kube-api-access-wqzth\") pod \"isvc-lightgbm-predictor-bdf964bd-lgk4l\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.480106 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.480077    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:26:19.483198 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.483179    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:19.610803 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.610772    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"]
May 25 13:26:19.615559 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:26:19.615525    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb626da8f_32c3_48e8_96a5_5e5c9efa2c98.slice/crio-0627390b76b851a5d4fd83c93bb063bcd13304c90c2f8aa5bf8b8fc64b71831d WatchSource:0}: Error finding container 0627390b76b851a5d4fd83c93bb063bcd13304c90c2f8aa5bf8b8fc64b71831d: Status 404 returned error can't find the container with id 0627390b76b851a5d4fd83c93bb063bcd13304c90c2f8aa5bf8b8fc64b71831d
May 25 13:26:19.616028 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.615998    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d598bc-f1f6-4998-993d-e750b6600165-message-dumper-kube-rbac-proxy-sar-config\") pod \"66d598bc-f1f6-4998-993d-e750b6600165\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") "
May 25 13:26:19.616101 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.616037    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls\") pod \"66d598bc-f1f6-4998-993d-e750b6600165\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") "
May 25 13:26:19.616152 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.616103    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb7mn\" (UniqueName: \"kubernetes.io/projected/66d598bc-f1f6-4998-993d-e750b6600165-kube-api-access-rb7mn\") pod \"66d598bc-f1f6-4998-993d-e750b6600165\" (UID: \"66d598bc-f1f6-4998-993d-e750b6600165\") "
May 25 13:26:19.616390 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.616366    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d598bc-f1f6-4998-993d-e750b6600165-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "66d598bc-f1f6-4998-993d-e750b6600165" (UID: "66d598bc-f1f6-4998-993d-e750b6600165"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:26:19.618258 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.618224    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66d598bc-f1f6-4998-993d-e750b6600165" (UID: "66d598bc-f1f6-4998-993d-e750b6600165"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:19.618403 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.618378    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d598bc-f1f6-4998-993d-e750b6600165-kube-api-access-rb7mn" (OuterVolumeSpecName: "kube-api-access-rb7mn") pod "66d598bc-f1f6-4998-993d-e750b6600165" (UID: "66d598bc-f1f6-4998-993d-e750b6600165"). InnerVolumeSpecName "kube-api-access-rb7mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:26:19.716958 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.716920    2549 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66d598bc-f1f6-4998-993d-e750b6600165-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:26:19.716958 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.716952    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66d598bc-f1f6-4998-993d-e750b6600165-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:26:19.716958 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:19.716965    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7mn\" (UniqueName: \"kubernetes.io/projected/66d598bc-f1f6-4998-993d-e750b6600165-kube-api-access-rb7mn\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:26:20.110308 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.110254    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" event={"ID":"b626da8f-32c3-48e8-96a5-5e5c9efa2c98","Type":"ContainerStarted","Data":"b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7"}
May 25 13:26:20.110754 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.110315    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" event={"ID":"b626da8f-32c3-48e8-96a5-5e5c9efa2c98","Type":"ContainerStarted","Data":"0627390b76b851a5d4fd83c93bb063bcd13304c90c2f8aa5bf8b8fc64b71831d"}
May 25 13:26:20.112601 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.112574    2549 generic.go:358] "Generic (PLEG): container finished" podID="555a3549-22bc-4739-811b-38f36ea8868a" containerID="f393fbf07816a1b35ee5c322b8f6e4b7a55d6031675a16864d37c1c3b46ae9bb" exitCode=2
May 25 13:26:20.112689 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.112644    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerDied","Data":"f393fbf07816a1b35ee5c322b8f6e4b7a55d6031675a16864d37c1c3b46ae9bb"}
May 25 13:26:20.114136 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.114116    2549 generic.go:358] "Generic (PLEG): container finished" podID="66d598bc-f1f6-4998-993d-e750b6600165" containerID="e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27" exitCode=2
May 25 13:26:20.114136 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.114135    2549 generic.go:358] "Generic (PLEG): container finished" podID="66d598bc-f1f6-4998-993d-e750b6600165" containerID="269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b" exitCode=2
May 25 13:26:20.114313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.114183    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" event={"ID":"66d598bc-f1f6-4998-993d-e750b6600165","Type":"ContainerDied","Data":"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27"}
May 25 13:26:20.114313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.114202    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"
May 25 13:26:20.114313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.114216    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" event={"ID":"66d598bc-f1f6-4998-993d-e750b6600165","Type":"ContainerDied","Data":"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b"}
May 25 13:26:20.114313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.114231    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8" event={"ID":"66d598bc-f1f6-4998-993d-e750b6600165","Type":"ContainerDied","Data":"30111adc5082aa8048a01c01194e3679448a70a4a64913e768b3d86037be6c83"}
May 25 13:26:20.114313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.114252    2549 scope.go:117] "RemoveContainer" containerID="e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27"
May 25 13:26:20.121759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.121740    2549 scope.go:117] "RemoveContainer" containerID="269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b"
May 25 13:26:20.129175 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.129160    2549 scope.go:117] "RemoveContainer" containerID="e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27"
May 25 13:26:20.129461 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:26:20.129436    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27\": container with ID starting with e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27 not found: ID does not exist" containerID="e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27"
May 25 13:26:20.129538 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.129473    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27"} err="failed to get container status \"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27\": rpc error: code = NotFound desc = could not find container \"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27\": container with ID starting with e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27 not found: ID does not exist"
May 25 13:26:20.129538 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.129500    2549 scope.go:117] "RemoveContainer" containerID="269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b"
May 25 13:26:20.129784 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:26:20.129760    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b\": container with ID starting with 269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b not found: ID does not exist" containerID="269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b"
May 25 13:26:20.129910 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.129788    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b"} err="failed to get container status \"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b\": rpc error: code = NotFound desc = could not find container \"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b\": container with ID starting with 269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b not found: ID does not exist"
May 25 13:26:20.129910 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.129807    2549 scope.go:117] "RemoveContainer" containerID="e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27"
May 25 13:26:20.130119 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.130083    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27"} err="failed to get container status \"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27\": rpc error: code = NotFound desc = could not find container \"e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27\": container with ID starting with e6c2b4ce78a12e4e38aab2d529f3f02f65e9c729992761573482b169420b0b27 not found: ID does not exist"
May 25 13:26:20.130119 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.130115    2549 scope.go:117] "RemoveContainer" containerID="269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b"
May 25 13:26:20.130651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.130463    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b"} err="failed to get container status \"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b\": rpc error: code = NotFound desc = could not find container \"269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b\": container with ID starting with 269c213c35646250963d4ee6840bd5a6a639cdbf49e12ac5e1ef037f9ed4f16b not found: ID does not exist"
May 25 13:26:20.141993 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.141964    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"]
May 25 13:26:20.145763 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.145739    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-ncqv8"]
May 25 13:26:20.496153 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:20.496121    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d598bc-f1f6-4998-993d-e750b6600165" path="/var/lib/kubelet/pods/66d598bc-f1f6-4998-993d-e750b6600165/volumes"
May 25 13:26:21.890810 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:21.890765    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 25 13:26:24.128346 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:24.128305    2549 generic.go:358] "Generic (PLEG): container finished" podID="555a3549-22bc-4739-811b-38f36ea8868a" containerID="fcf178c67ad7df368fef4a63f0e0fbe5421903ea35786d3b77a1e6cd542294d0" exitCode=0
May 25 13:26:24.128745 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:24.128362    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerDied","Data":"fcf178c67ad7df368fef4a63f0e0fbe5421903ea35786d3b77a1e6cd542294d0"}
May 25 13:26:24.129686 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:24.129666    2549 generic.go:358] "Generic (PLEG): container finished" podID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerID="b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7" exitCode=0
May 25 13:26:24.129769 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:24.129701    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" event={"ID":"b626da8f-32c3-48e8-96a5-5e5c9efa2c98","Type":"ContainerDied","Data":"b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7"}
May 25 13:26:26.884263 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:26.884214    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 25 13:26:26.888572 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:26.888532    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:26:26.888976 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:26.888942    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:31.154854 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:31.154810    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" event={"ID":"b626da8f-32c3-48e8-96a5-5e5c9efa2c98","Type":"ContainerStarted","Data":"3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811"}
May 25 13:26:31.155240 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:31.154866    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" event={"ID":"b626da8f-32c3-48e8-96a5-5e5c9efa2c98","Type":"ContainerStarted","Data":"1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec"}
May 25 13:26:31.155240 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:31.155085    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:31.176093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:31.176047    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podStartSLOduration=5.96189488 podStartE2EDuration="12.176034416s" podCreationTimestamp="2026-05-25 13:26:19 +0000 UTC" firstStartedPulling="2026-05-25 13:26:24.130872167 +0000 UTC m=+1334.169629269" lastFinishedPulling="2026-05-25 13:26:30.3450117 +0000 UTC m=+1340.383768805" observedRunningTime="2026-05-25 13:26:31.174877849 +0000 UTC m=+1341.213634972" watchObservedRunningTime="2026-05-25 13:26:31.176034416 +0000 UTC m=+1341.214791540"
May 25 13:26:31.883906 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:31.883863    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 25 13:26:31.884087 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:31.883990    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:26:32.157513 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:32.157427    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:32.158558 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:32.158529    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:26:33.160365 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:33.160319    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:26:36.884592 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:36.884548    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 25 13:26:36.888957 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:36.888917    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:26:36.889249 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:36.889228    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:38.165546 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:38.165515    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:26:38.166088 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:38.166058    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:26:41.884466 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:41.884420    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 25 13:26:46.884599 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:46.884557    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 25 13:26:46.888892 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:46.888848    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
May 25 13:26:46.889040 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:46.888997    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:26:46.889101 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:46.889082    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:26:46.889201 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:46.889176    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:26:48.166680 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:48.166641    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:26:49.213331 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.213293    2549 generic.go:358] "Generic (PLEG): container finished" podID="555a3549-22bc-4739-811b-38f36ea8868a" containerID="dc0f09ec85270489a8de9a901a046bb50315b180ebb894f7bf980576aa146325" exitCode=0
May 25 13:26:49.213696 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.213364    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerDied","Data":"dc0f09ec85270489a8de9a901a046bb50315b180ebb894f7bf980576aa146325"}
May 25 13:26:49.322902 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.322876    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:26:49.460346 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.460239    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/555a3549-22bc-4739-811b-38f36ea8868a-isvc-logger-kube-rbac-proxy-sar-config\") pod \"555a3549-22bc-4739-811b-38f36ea8868a\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") "
May 25 13:26:49.460346 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.460307    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/555a3549-22bc-4739-811b-38f36ea8868a-kserve-provision-location\") pod \"555a3549-22bc-4739-811b-38f36ea8868a\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") "
May 25 13:26:49.460346 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.460336    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw8nc\" (UniqueName: \"kubernetes.io/projected/555a3549-22bc-4739-811b-38f36ea8868a-kube-api-access-cw8nc\") pod \"555a3549-22bc-4739-811b-38f36ea8868a\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") "
May 25 13:26:49.460630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.460428    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls\") pod \"555a3549-22bc-4739-811b-38f36ea8868a\" (UID: \"555a3549-22bc-4739-811b-38f36ea8868a\") "
May 25 13:26:49.460721 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.460690    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555a3549-22bc-4739-811b-38f36ea8868a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "555a3549-22bc-4739-811b-38f36ea8868a" (UID: "555a3549-22bc-4739-811b-38f36ea8868a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:26:49.460769 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.460712    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555a3549-22bc-4739-811b-38f36ea8868a-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "555a3549-22bc-4739-811b-38f36ea8868a" (UID: "555a3549-22bc-4739-811b-38f36ea8868a"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:26:49.462543 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.462513    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "555a3549-22bc-4739-811b-38f36ea8868a" (UID: "555a3549-22bc-4739-811b-38f36ea8868a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:49.462820 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.462801    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555a3549-22bc-4739-811b-38f36ea8868a-kube-api-access-cw8nc" (OuterVolumeSpecName: "kube-api-access-cw8nc") pod "555a3549-22bc-4739-811b-38f36ea8868a" (UID: "555a3549-22bc-4739-811b-38f36ea8868a"). InnerVolumeSpecName "kube-api-access-cw8nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:26:49.561086 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.561047    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/555a3549-22bc-4739-811b-38f36ea8868a-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:26:49.561086 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.561075    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/555a3549-22bc-4739-811b-38f36ea8868a-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:26:49.561086 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.561085    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/555a3549-22bc-4739-811b-38f36ea8868a-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:26:49.561086 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:49.561096    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8nc\" (UniqueName: \"kubernetes.io/projected/555a3549-22bc-4739-811b-38f36ea8868a-kube-api-access-cw8nc\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:26:50.218538 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.218505    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7" event={"ID":"555a3549-22bc-4739-811b-38f36ea8868a","Type":"ContainerDied","Data":"54e7d672ffa6ab3d7a85bb868e717da1d3e01aed5807a8849768b4d903ca2f4a"}
May 25 13:26:50.219027 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.218552    2549 scope.go:117] "RemoveContainer" containerID="dc0f09ec85270489a8de9a901a046bb50315b180ebb894f7bf980576aa146325"
May 25 13:26:50.219027 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.218587    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"
May 25 13:26:50.227301 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.227256    2549 scope.go:117] "RemoveContainer" containerID="f393fbf07816a1b35ee5c322b8f6e4b7a55d6031675a16864d37c1c3b46ae9bb"
May 25 13:26:50.234670 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.234651    2549 scope.go:117] "RemoveContainer" containerID="fcf178c67ad7df368fef4a63f0e0fbe5421903ea35786d3b77a1e6cd542294d0"
May 25 13:26:50.241293 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.241239    2549 scope.go:117] "RemoveContainer" containerID="e9a5eaefb27b6963893b9ee2bd62a94c39c7f2a37a1b5f710236f3937b0d5cd6"
May 25 13:26:50.249706 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.249681    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"]
May 25 13:26:50.253950 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.253927    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-7xvh7"]
May 25 13:26:50.495562 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:50.495484    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="555a3549-22bc-4739-811b-38f36ea8868a" path="/var/lib/kubelet/pods/555a3549-22bc-4739-811b-38f36ea8868a/volumes"
May 25 13:26:58.166563 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:26:58.166523    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:27:08.166450 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:08.166405    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:27:18.166612 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:18.166570    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:27:28.166377 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:28.166319    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:27:38.166869 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:38.166825    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 25 13:27:48.166920 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:48.166887    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:27:49.293014 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.292976    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"]
May 25 13:27:49.293440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.293317    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container" containerID="cri-o://1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec" gracePeriod=30
May 25 13:27:49.293497 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.293397    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kube-rbac-proxy" containerID="cri-o://3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811" gracePeriod=30
May 25 13:27:49.455307 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455248    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"]
May 25 13:27:49.455577 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455564    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent"
May 25 13:27:49.455630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455580    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent"
May 25 13:27:49.455630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455591    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d598bc-f1f6-4998-993d-e750b6600165" containerName="kserve-container"
May 25 13:27:49.455630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455601    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d598bc-f1f6-4998-993d-e750b6600165" containerName="kserve-container"
May 25 13:27:49.455630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455610    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy"
May 25 13:27:49.455630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455616    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy"
May 25 13:27:49.455630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455625    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d598bc-f1f6-4998-993d-e750b6600165" containerName="kube-rbac-proxy"
May 25 13:27:49.455630 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455630    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d598bc-f1f6-4998-993d-e750b6600165" containerName="kube-rbac-proxy"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455638    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455643    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455650    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="storage-initializer"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455655    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="storage-initializer"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455703    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="agent"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455711    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kserve-container"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455718    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="66d598bc-f1f6-4998-993d-e750b6600165" containerName="kube-rbac-proxy"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455728    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="555a3549-22bc-4739-811b-38f36ea8868a" containerName="kube-rbac-proxy"
May 25 13:27:49.455894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.455734    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="66d598bc-f1f6-4998-993d-e750b6600165" containerName="kserve-container"
May 25 13:27:49.458619 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.458597    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.460945 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.460918    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
May 25 13:27:49.461074 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.460923    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
May 25 13:27:49.467851 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.467820    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"]
May 25 13:27:49.608891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.608789    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed7845f-fe0f-4608-85aa-0de3a6bcde44-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.609059 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.608952    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.609116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.609071    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.609116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.609112    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm25n\" (UniqueName: \"kubernetes.io/projected/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kube-api-access-mm25n\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.710176 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.710121    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.710403 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.710189    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mm25n\" (UniqueName: \"kubernetes.io/projected/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kube-api-access-mm25n\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.710403 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.710231    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed7845f-fe0f-4608-85aa-0de3a6bcde44-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.710403 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.710297    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.710570 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:27:49.710420    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 25 13:27:49.710570 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:27:49.710493    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls podName:fed7845f-fe0f-4608-85aa-0de3a6bcde44 nodeName:}" failed. No retries permitted until 2026-05-25 13:27:50.210471532 +0000 UTC m=+1420.249228637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls") pod "isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" (UID: "fed7845f-fe0f-4608-85aa-0de3a6bcde44") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 25 13:27:49.710677 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.710618    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.710920 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.710901    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed7845f-fe0f-4608-85aa-0de3a6bcde44-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:49.719672 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:49.719642    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm25n\" (UniqueName: \"kubernetes.io/projected/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kube-api-access-mm25n\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:50.215138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:50.215101    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:50.217733 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:50.217703    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:50.369946 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:50.369902    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:50.384007 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:50.383974    2549 generic.go:358] "Generic (PLEG): container finished" podID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerID="3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811" exitCode=2
May 25 13:27:50.384164 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:50.384032    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" event={"ID":"b626da8f-32c3-48e8-96a5-5e5c9efa2c98","Type":"ContainerDied","Data":"3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811"}
May 25 13:27:50.510440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:50.510412    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"]
May 25 13:27:50.513055 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:27:50.513020    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed7845f_fe0f_4608_85aa_0de3a6bcde44.slice/crio-1ddca7d0065a9f8b32cfad6fd8b441ad28204eafba5e9da74d3a76daa56a8531 WatchSource:0}: Error finding container 1ddca7d0065a9f8b32cfad6fd8b441ad28204eafba5e9da74d3a76daa56a8531: Status 404 returned error can't find the container with id 1ddca7d0065a9f8b32cfad6fd8b441ad28204eafba5e9da74d3a76daa56a8531
May 25 13:27:51.388064 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:51.388021    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" event={"ID":"fed7845f-fe0f-4608-85aa-0de3a6bcde44","Type":"ContainerStarted","Data":"21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5"}
May 25 13:27:51.388064 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:51.388069    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" event={"ID":"fed7845f-fe0f-4608-85aa-0de3a6bcde44","Type":"ContainerStarted","Data":"1ddca7d0065a9f8b32cfad6fd8b441ad28204eafba5e9da74d3a76daa56a8531"}
May 25 13:27:53.160638 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:53.160582    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 25 13:27:54.398042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:54.398008    2549 generic.go:358] "Generic (PLEG): container finished" podID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerID="21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5" exitCode=0
May 25 13:27:54.398445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:54.398090    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" event={"ID":"fed7845f-fe0f-4608-85aa-0de3a6bcde44","Type":"ContainerDied","Data":"21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5"}
May 25 13:27:55.238624 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.238588    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:27:55.357594 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.357507    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqzth\" (UniqueName: \"kubernetes.io/projected/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kube-api-access-wqzth\") pod \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") "
May 25 13:27:55.357781 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.357604    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") "
May 25 13:27:55.357781 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.357639    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-proxy-tls\") pod \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") "
May 25 13:27:55.357781 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.357656    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kserve-provision-location\") pod \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\" (UID: \"b626da8f-32c3-48e8-96a5-5e5c9efa2c98\") "
May 25 13:27:55.358022 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.357998    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b626da8f-32c3-48e8-96a5-5e5c9efa2c98" (UID: "b626da8f-32c3-48e8-96a5-5e5c9efa2c98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:27:55.358076 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.357996    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "b626da8f-32c3-48e8-96a5-5e5c9efa2c98" (UID: "b626da8f-32c3-48e8-96a5-5e5c9efa2c98"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:27:55.359688 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.359662    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kube-api-access-wqzth" (OuterVolumeSpecName: "kube-api-access-wqzth") pod "b626da8f-32c3-48e8-96a5-5e5c9efa2c98" (UID: "b626da8f-32c3-48e8-96a5-5e5c9efa2c98"). InnerVolumeSpecName "kube-api-access-wqzth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:27:55.359789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.359768    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b626da8f-32c3-48e8-96a5-5e5c9efa2c98" (UID: "b626da8f-32c3-48e8-96a5-5e5c9efa2c98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:27:55.403081 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.403045    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" event={"ID":"fed7845f-fe0f-4608-85aa-0de3a6bcde44","Type":"ContainerStarted","Data":"de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3"}
May 25 13:27:55.403081 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.403095    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" event={"ID":"fed7845f-fe0f-4608-85aa-0de3a6bcde44","Type":"ContainerStarted","Data":"c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb"}
May 25 13:27:55.403599 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.403401    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:55.403599 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.403564    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:27:55.404849 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.404820    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:27:55.404983 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.404889    2549 generic.go:358] "Generic (PLEG): container finished" podID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerID="1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec" exitCode=0
May 25 13:27:55.404983 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.404930    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" event={"ID":"b626da8f-32c3-48e8-96a5-5e5c9efa2c98","Type":"ContainerDied","Data":"1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec"}
May 25 13:27:55.404983 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.404955    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l" event={"ID":"b626da8f-32c3-48e8-96a5-5e5c9efa2c98","Type":"ContainerDied","Data":"0627390b76b851a5d4fd83c93bb063bcd13304c90c2f8aa5bf8b8fc64b71831d"}
May 25 13:27:55.404983 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.404972    2549 scope.go:117] "RemoveContainer" containerID="3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811"
May 25 13:27:55.405166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.405042    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"
May 25 13:27:55.413010 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.412986    2549 scope.go:117] "RemoveContainer" containerID="1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec"
May 25 13:27:55.420284 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.420250    2549 scope.go:117] "RemoveContainer" containerID="b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7"
May 25 13:27:55.424358 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.424308    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podStartSLOduration=6.424260069 podStartE2EDuration="6.424260069s" podCreationTimestamp="2026-05-25 13:27:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:27:55.421962399 +0000 UTC m=+1425.460719522" watchObservedRunningTime="2026-05-25 13:27:55.424260069 +0000 UTC m=+1425.463017192"
May 25 13:27:55.427885 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.427868    2549 scope.go:117] "RemoveContainer" containerID="3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811"
May 25 13:27:55.428163 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:27:55.428145    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811\": container with ID starting with 3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811 not found: ID does not exist" containerID="3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811"
May 25 13:27:55.428207 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.428173    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811"} err="failed to get container status \"3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811\": rpc error: code = NotFound desc = could not find container \"3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811\": container with ID starting with 3ca91fecc46b8bd65036fc37f4e0898c5dba894f9cb4a15f78a2f295de8c1811 not found: ID does not exist"
May 25 13:27:55.428207 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.428189    2549 scope.go:117] "RemoveContainer" containerID="1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec"
May 25 13:27:55.428464 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:27:55.428445    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec\": container with ID starting with 1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec not found: ID does not exist" containerID="1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec"
May 25 13:27:55.428508 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.428471    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec"} err="failed to get container status \"1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec\": rpc error: code = NotFound desc = could not find container \"1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec\": container with ID starting with 1e042a6c40f73323a385ff5046d2dcc026289fd75244f298d9a67054d95894ec not found: ID does not exist"
May 25 13:27:55.428508 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.428493    2549 scope.go:117] "RemoveContainer" containerID="b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7"
May 25 13:27:55.428738 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:27:55.428719    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7\": container with ID starting with b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7 not found: ID does not exist" containerID="b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7"
May 25 13:27:55.428791 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.428743    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7"} err="failed to get container status \"b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7\": rpc error: code = NotFound desc = could not find container \"b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7\": container with ID starting with b518e220c9c73a06380a0d5c18dfc6f228d75007fe8ebba266111e756351b2d7 not found: ID does not exist"
May 25 13:27:55.440532 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.440493    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"]
May 25 13:27:55.447213 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.447182    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-lgk4l"]
May 25 13:27:55.458472 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.458447    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:27:55.458602 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.458474    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:27:55.458602 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.458487    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:27:55.458602 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:55.458498    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqzth\" (UniqueName: \"kubernetes.io/projected/b626da8f-32c3-48e8-96a5-5e5c9efa2c98-kube-api-access-wqzth\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:27:56.408925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:56.408883    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:27:56.499527 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:27:56.499492    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" path="/var/lib/kubelet/pods/b626da8f-32c3-48e8-96a5-5e5c9efa2c98/volumes"
May 25 13:28:01.413979 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:28:01.413946    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:28:01.414491 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:28:01.414464    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:28:11.415376 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:28:11.415333    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:28:21.414564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:28:21.414525    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:28:31.414472 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:28:31.414429    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:28:41.414761 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:28:41.414717    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:28:51.414521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:28:51.414475    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:29:01.415449 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:01.415410    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:29:07.492474 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:07.492428    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 25 13:29:11.082520 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:11.082487    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:29:11.084665 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:11.084645    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:29:17.493004 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:17.492973    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:29:20.025978 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.025891    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"]
May 25 13:29:20.026387 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.026195    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container" containerID="cri-o://c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb" gracePeriod=30
May 25 13:29:20.026387 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.026244    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kube-rbac-proxy" containerID="cri-o://de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3" gracePeriod=30
May 25 13:29:20.135250 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135219    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"]
May 25 13:29:20.135511 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135498    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container"
May 25 13:29:20.135554 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135515    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container"
May 25 13:29:20.135554 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135536    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kube-rbac-proxy"
May 25 13:29:20.135554 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135541    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kube-rbac-proxy"
May 25 13:29:20.135554 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135549    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="storage-initializer"
May 25 13:29:20.135554 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135555    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="storage-initializer"
May 25 13:29:20.135699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135608    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kube-rbac-proxy"
May 25 13:29:20.135699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.135621    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="b626da8f-32c3-48e8-96a5-5e5c9efa2c98" containerName="kserve-container"
May 25 13:29:20.138603 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.138586    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.141342 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.141319    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
May 25 13:29:20.141469 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.141450    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 13:29:20.148965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.148944    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"]
May 25 13:29:20.304731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.304633    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e143ee6-df97-41c0-b597-e3638c591e52-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.304731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.304673    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e143ee6-df97-41c0-b597-e3638c591e52-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.304731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.304697    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r74\" (UniqueName: \"kubernetes.io/projected/0e143ee6-df97-41c0-b597-e3638c591e52-kube-api-access-67r74\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.304971 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.304768    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e143ee6-df97-41c0-b597-e3638c591e52-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.405958 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.405900    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e143ee6-df97-41c0-b597-e3638c591e52-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.405958 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.405966    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e143ee6-df97-41c0-b597-e3638c591e52-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.406227 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.406001    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-67r74\" (UniqueName: \"kubernetes.io/projected/0e143ee6-df97-41c0-b597-e3638c591e52-kube-api-access-67r74\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.406227 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.406034    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e143ee6-df97-41c0-b597-e3638c591e52-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.406525 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.406503    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e143ee6-df97-41c0-b597-e3638c591e52-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.406708 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.406685    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e143ee6-df97-41c0-b597-e3638c591e52-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.408540 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.408522    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e143ee6-df97-41c0-b597-e3638c591e52-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.424177 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.424151    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r74\" (UniqueName: \"kubernetes.io/projected/0e143ee6-df97-41c0-b597-e3638c591e52-kube-api-access-67r74\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.448406 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.448380    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:29:20.579474 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.579448    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"]
May 25 13:29:20.581663 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:29:20.581628    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e143ee6_df97_41c0_b597_e3638c591e52.slice/crio-b23ded0f0ee09e667f399f454b706d4e80147b378775b96d8324dcc11494d1f9 WatchSource:0}: Error finding container b23ded0f0ee09e667f399f454b706d4e80147b378775b96d8324dcc11494d1f9: Status 404 returned error can't find the container with id b23ded0f0ee09e667f399f454b706d4e80147b378775b96d8324dcc11494d1f9
May 25 13:29:20.583369 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.583353    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:29:20.632215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.632185    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" event={"ID":"0e143ee6-df97-41c0-b597-e3638c591e52","Type":"ContainerStarted","Data":"b23ded0f0ee09e667f399f454b706d4e80147b378775b96d8324dcc11494d1f9"}
May 25 13:29:20.634264 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.634243    2549 generic.go:358] "Generic (PLEG): container finished" podID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerID="de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3" exitCode=2
May 25 13:29:20.634374 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:20.634298    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" event={"ID":"fed7845f-fe0f-4608-85aa-0de3a6bcde44","Type":"ContainerDied","Data":"de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3"}
May 25 13:29:21.409753 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:21.409702    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 25 13:29:21.638560 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:21.638517    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" event={"ID":"0e143ee6-df97-41c0-b597-e3638c591e52","Type":"ContainerStarted","Data":"9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a"}
May 25 13:29:24.647181 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:24.647145    2549 generic.go:358] "Generic (PLEG): container finished" podID="0e143ee6-df97-41c0-b597-e3638c591e52" containerID="9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a" exitCode=0
May 25 13:29:24.647577 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:24.647223    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" event={"ID":"0e143ee6-df97-41c0-b597-e3638c591e52","Type":"ContainerDied","Data":"9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a"}
May 25 13:29:25.078694 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.078653    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:29:25.246563 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.246485    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls\") pod \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") "
May 25 13:29:25.246563 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.246547    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed7845f-fe0f-4608-85aa-0de3a6bcde44-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") "
May 25 13:29:25.246788 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.246591    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm25n\" (UniqueName: \"kubernetes.io/projected/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kube-api-access-mm25n\") pod \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") "
May 25 13:29:25.246788 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.246654    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kserve-provision-location\") pod \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\" (UID: \"fed7845f-fe0f-4608-85aa-0de3a6bcde44\") "
May 25 13:29:25.247773 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.247739    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed7845f-fe0f-4608-85aa-0de3a6bcde44-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "fed7845f-fe0f-4608-85aa-0de3a6bcde44" (UID: "fed7845f-fe0f-4608-85aa-0de3a6bcde44"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:29:25.247949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.247926    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fed7845f-fe0f-4608-85aa-0de3a6bcde44" (UID: "fed7845f-fe0f-4608-85aa-0de3a6bcde44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:29:25.251042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.251006    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kube-api-access-mm25n" (OuterVolumeSpecName: "kube-api-access-mm25n") pod "fed7845f-fe0f-4608-85aa-0de3a6bcde44" (UID: "fed7845f-fe0f-4608-85aa-0de3a6bcde44"). InnerVolumeSpecName "kube-api-access-mm25n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:29:25.252819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.252790    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fed7845f-fe0f-4608-85aa-0de3a6bcde44" (UID: "fed7845f-fe0f-4608-85aa-0de3a6bcde44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:29:25.347775 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.347732    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:29:25.347775 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.347777    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed7845f-fe0f-4608-85aa-0de3a6bcde44-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:29:25.348012 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.347797    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed7845f-fe0f-4608-85aa-0de3a6bcde44-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:29:25.348012 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.347813    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm25n\" (UniqueName: \"kubernetes.io/projected/fed7845f-fe0f-4608-85aa-0de3a6bcde44-kube-api-access-mm25n\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:29:25.659233 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.658994    2549 generic.go:358] "Generic (PLEG): container finished" podID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerID="c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb" exitCode=0
May 25 13:29:25.659233 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.659088    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" event={"ID":"fed7845f-fe0f-4608-85aa-0de3a6bcde44","Type":"ContainerDied","Data":"c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb"}
May 25 13:29:25.659233 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.659115    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"
May 25 13:29:25.659233 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.659132    2549 scope.go:117] "RemoveContainer" containerID="de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3"
May 25 13:29:25.660287 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.659120    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g" event={"ID":"fed7845f-fe0f-4608-85aa-0de3a6bcde44","Type":"ContainerDied","Data":"1ddca7d0065a9f8b32cfad6fd8b441ad28204eafba5e9da74d3a76daa56a8531"}
May 25 13:29:25.674141 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.673809    2549 scope.go:117] "RemoveContainer" containerID="c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb"
May 25 13:29:25.688151 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.687311    2549 scope.go:117] "RemoveContainer" containerID="21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5"
May 25 13:29:25.689870 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.689828    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"]
May 25 13:29:25.692801 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.692602    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-2pf7g"]
May 25 13:29:25.702410 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.702391    2549 scope.go:117] "RemoveContainer" containerID="de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3"
May 25 13:29:25.702804 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:29:25.702751    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3\": container with ID starting with de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3 not found: ID does not exist" containerID="de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3"
May 25 13:29:25.702804 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.702789    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3"} err="failed to get container status \"de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3\": rpc error: code = NotFound desc = could not find container \"de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3\": container with ID starting with de0a2eed32ced3077b1528b3550a3cf6b5c66343e05140dd95d9809e072f1be3 not found: ID does not exist"
May 25 13:29:25.702948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.702811    2549 scope.go:117] "RemoveContainer" containerID="c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb"
May 25 13:29:25.703151 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:29:25.703122    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb\": container with ID starting with c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb not found: ID does not exist" containerID="c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb"
May 25 13:29:25.703215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.703162    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb"} err="failed to get container status \"c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb\": rpc error: code = NotFound desc = could not find container \"c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb\": container with ID starting with c2c645c39ba4e81f13f9fe9c8bbe4598fe356ed7b8f2f42a82bc9814319ea1eb not found: ID does not exist"
May 25 13:29:25.703215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.703187    2549 scope.go:117] "RemoveContainer" containerID="21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5"
May 25 13:29:25.704053 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:29:25.704026    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5\": container with ID starting with 21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5 not found: ID does not exist" containerID="21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5"
May 25 13:29:25.704125 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:25.704063    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5"} err="failed to get container status \"21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5\": rpc error: code = NotFound desc = could not find container \"21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5\": container with ID starting with 21a0f5ca237810e01463b6ba9a6feb8dec3ae5a72a9b69a9c5f4c48e32fc69c5 not found: ID does not exist"
May 25 13:29:26.500545 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:29:26.500075    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" path="/var/lib/kubelet/pods/fed7845f-fe0f-4608-85aa-0de3a6bcde44/volumes"
May 25 13:31:37.051038 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:31:37.051002    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" event={"ID":"0e143ee6-df97-41c0-b597-e3638c591e52","Type":"ContainerStarted","Data":"231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b"}
May 25 13:31:37.051038 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:31:37.051044    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" event={"ID":"0e143ee6-df97-41c0-b597-e3638c591e52","Type":"ContainerStarted","Data":"242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684"}
May 25 13:31:37.051503 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:31:37.051244    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:31:37.051503 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:31:37.051288    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:31:37.081329 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:31:37.081204    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" podStartSLOduration=4.921769939 podStartE2EDuration="2m17.081188599s" podCreationTimestamp="2026-05-25 13:29:20 +0000 UTC" firstStartedPulling="2026-05-25 13:29:24.648350105 +0000 UTC m=+1514.687107209" lastFinishedPulling="2026-05-25 13:31:36.807768767 +0000 UTC m=+1646.846525869" observedRunningTime="2026-05-25 13:31:37.079128586 +0000 UTC m=+1647.117885712" watchObservedRunningTime="2026-05-25 13:31:37.081188599 +0000 UTC m=+1647.119945722"
May 25 13:31:43.058676 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:31:43.058637    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:32:13.062750 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:13.062718    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:32:20.260820 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.260737    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"]
May 25 13:32:20.261340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.261047    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="kserve-container" containerID="cri-o://242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684" gracePeriod=30
May 25 13:32:20.261340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.261095    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="kube-rbac-proxy" containerID="cri-o://231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b" gracePeriod=30
May 25 13:32:20.496358 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496320    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"]
May 25 13:32:20.496609 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496595    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="storage-initializer"
May 25 13:32:20.496651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496611    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="storage-initializer"
May 25 13:32:20.496651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496620    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container"
May 25 13:32:20.496651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496626    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container"
May 25 13:32:20.496651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496640    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kube-rbac-proxy"
May 25 13:32:20.496651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496645    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kube-rbac-proxy"
May 25 13:32:20.496796 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496689    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kube-rbac-proxy"
May 25 13:32:20.496796 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.496699    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="fed7845f-fe0f-4608-85aa-0de3a6bcde44" containerName="kserve-container"
May 25 13:32:20.498515 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.498498    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.500832 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.500810    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 13:32:20.500949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.500907    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
May 25 13:32:20.511431 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.511369    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"]
May 25 13:32:20.578965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.578911    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf79461-bb62-4c85-8266-401065829631-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.579151 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.578991    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf79461-bb62-4c85-8266-401065829631-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.579151 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.579070    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf79461-bb62-4c85-8266-401065829631-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.579151 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.579100    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgp4v\" (UniqueName: \"kubernetes.io/projected/ccf79461-bb62-4c85-8266-401065829631-kube-api-access-jgp4v\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.679791 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.679745    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf79461-bb62-4c85-8266-401065829631-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.679980 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.679805    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf79461-bb62-4c85-8266-401065829631-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.679980 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.679848    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf79461-bb62-4c85-8266-401065829631-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.679980 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.679879    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgp4v\" (UniqueName: \"kubernetes.io/projected/ccf79461-bb62-4c85-8266-401065829631-kube-api-access-jgp4v\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.680329 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.680300    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf79461-bb62-4c85-8266-401065829631-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.680539 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.680518    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf79461-bb62-4c85-8266-401065829631-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.682372 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.682356    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf79461-bb62-4c85-8266-401065829631-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.688050 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.688027    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgp4v\" (UniqueName: \"kubernetes.io/projected/ccf79461-bb62-4c85-8266-401065829631-kube-api-access-jgp4v\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.811051 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.810958    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:20.945730 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:20.945693    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"]
May 25 13:32:20.948739 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:32:20.948706    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf79461_bb62_4c85_8266_401065829631.slice/crio-06bd9582126d18778387608878fff9a490ee1b60bf99fa4bc67474a4de9b5ac8 WatchSource:0}: Error finding container 06bd9582126d18778387608878fff9a490ee1b60bf99fa4bc67474a4de9b5ac8: Status 404 returned error can't find the container with id 06bd9582126d18778387608878fff9a490ee1b60bf99fa4bc67474a4de9b5ac8
May 25 13:32:21.173405 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.173373    2549 generic.go:358] "Generic (PLEG): container finished" podID="0e143ee6-df97-41c0-b597-e3638c591e52" containerID="231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b" exitCode=2
May 25 13:32:21.173617 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.173455    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" event={"ID":"0e143ee6-df97-41c0-b597-e3638c591e52","Type":"ContainerDied","Data":"231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b"}
May 25 13:32:21.174869 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.174836    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" event={"ID":"ccf79461-bb62-4c85-8266-401065829631","Type":"ContainerStarted","Data":"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace"}
May 25 13:32:21.174869 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.174863    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" event={"ID":"ccf79461-bb62-4c85-8266-401065829631","Type":"ContainerStarted","Data":"06bd9582126d18778387608878fff9a490ee1b60bf99fa4bc67474a4de9b5ac8"}
May 25 13:32:21.409418 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.409393    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:32:21.487780 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.487695    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e143ee6-df97-41c0-b597-e3638c591e52-proxy-tls\") pod \"0e143ee6-df97-41c0-b597-e3638c591e52\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") "
May 25 13:32:21.487780 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.487738    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e143ee6-df97-41c0-b597-e3638c591e52-kserve-provision-location\") pod \"0e143ee6-df97-41c0-b597-e3638c591e52\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") "
May 25 13:32:21.487780 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.487763    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e143ee6-df97-41c0-b597-e3638c591e52-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"0e143ee6-df97-41c0-b597-e3638c591e52\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") "
May 25 13:32:21.488002 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.487799    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67r74\" (UniqueName: \"kubernetes.io/projected/0e143ee6-df97-41c0-b597-e3638c591e52-kube-api-access-67r74\") pod \"0e143ee6-df97-41c0-b597-e3638c591e52\" (UID: \"0e143ee6-df97-41c0-b597-e3638c591e52\") "
May 25 13:32:21.488142 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.488115    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e143ee6-df97-41c0-b597-e3638c591e52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e143ee6-df97-41c0-b597-e3638c591e52" (UID: "0e143ee6-df97-41c0-b597-e3638c591e52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:32:21.488142 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.488129    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e143ee6-df97-41c0-b597-e3638c591e52-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "0e143ee6-df97-41c0-b597-e3638c591e52" (UID: "0e143ee6-df97-41c0-b597-e3638c591e52"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:32:21.490042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.490012    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e143ee6-df97-41c0-b597-e3638c591e52-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e143ee6-df97-41c0-b597-e3638c591e52" (UID: "0e143ee6-df97-41c0-b597-e3638c591e52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:32:21.490126 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.490040    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e143ee6-df97-41c0-b597-e3638c591e52-kube-api-access-67r74" (OuterVolumeSpecName: "kube-api-access-67r74") pod "0e143ee6-df97-41c0-b597-e3638c591e52" (UID: "0e143ee6-df97-41c0-b597-e3638c591e52"). InnerVolumeSpecName "kube-api-access-67r74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:32:21.588782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.588743    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e143ee6-df97-41c0-b597-e3638c591e52-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:32:21.588782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.588778    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e143ee6-df97-41c0-b597-e3638c591e52-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:32:21.588782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.588788    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e143ee6-df97-41c0-b597-e3638c591e52-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:32:21.588782 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:21.588799    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67r74\" (UniqueName: \"kubernetes.io/projected/0e143ee6-df97-41c0-b597-e3638c591e52-kube-api-access-67r74\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:32:22.179780 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.179742    2549 generic.go:358] "Generic (PLEG): container finished" podID="0e143ee6-df97-41c0-b597-e3638c591e52" containerID="242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684" exitCode=0
May 25 13:32:22.179938 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.179827    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"
May 25 13:32:22.179938 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.179830    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" event={"ID":"0e143ee6-df97-41c0-b597-e3638c591e52","Type":"ContainerDied","Data":"242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684"}
May 25 13:32:22.179938 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.179868    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph" event={"ID":"0e143ee6-df97-41c0-b597-e3638c591e52","Type":"ContainerDied","Data":"b23ded0f0ee09e667f399f454b706d4e80147b378775b96d8324dcc11494d1f9"}
May 25 13:32:22.179938 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.179888    2549 scope.go:117] "RemoveContainer" containerID="231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b"
May 25 13:32:22.188994 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.188977    2549 scope.go:117] "RemoveContainer" containerID="242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684"
May 25 13:32:22.196547 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.196526    2549 scope.go:117] "RemoveContainer" containerID="9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a"
May 25 13:32:22.203578 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.203560    2549 scope.go:117] "RemoveContainer" containerID="231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b"
May 25 13:32:22.203835 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:32:22.203818    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b\": container with ID starting with 231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b not found: ID does not exist" containerID="231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b"
May 25 13:32:22.203882 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.203842    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b"} err="failed to get container status \"231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b\": rpc error: code = NotFound desc = could not find container \"231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b\": container with ID starting with 231795de5d8daafd8003d121ff2ab412f9f224492378aed91831865f1016aa0b not found: ID does not exist"
May 25 13:32:22.203882 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.203861    2549 scope.go:117] "RemoveContainer" containerID="242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684"
May 25 13:32:22.204084 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:32:22.204067    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684\": container with ID starting with 242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684 not found: ID does not exist" containerID="242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684"
May 25 13:32:22.204123 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.204091    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684"} err="failed to get container status \"242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684\": rpc error: code = NotFound desc = could not find container \"242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684\": container with ID starting with 242d3e97c3ee0f422365badc5536499979bfcc3dc8feb85ed6c60a9707f9b684 not found: ID does not exist"
May 25 13:32:22.204123 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.204108    2549 scope.go:117] "RemoveContainer" containerID="9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a"
May 25 13:32:22.204361 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:32:22.204343    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a\": container with ID starting with 9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a not found: ID does not exist" containerID="9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a"
May 25 13:32:22.204401 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.204367    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a"} err="failed to get container status \"9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a\": rpc error: code = NotFound desc = could not find container \"9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a\": container with ID starting with 9c9ba524a3d218f9ab8986c20605784f0362e36e7d60897b04b46d7740232b0a not found: ID does not exist"
May 25 13:32:22.214350 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.214325    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"]
May 25 13:32:22.222223 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.222197    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-vjkph"]
May 25 13:32:22.495363 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:22.495255    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" path="/var/lib/kubelet/pods/0e143ee6-df97-41c0-b597-e3638c591e52/volumes"
May 25 13:32:25.195589 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:25.195503    2549 generic.go:358] "Generic (PLEG): container finished" podID="ccf79461-bb62-4c85-8266-401065829631" containerID="1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace" exitCode=0
May 25 13:32:25.195589 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:25.195559    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" event={"ID":"ccf79461-bb62-4c85-8266-401065829631","Type":"ContainerDied","Data":"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace"}
May 25 13:32:26.201120 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:26.201083    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" event={"ID":"ccf79461-bb62-4c85-8266-401065829631","Type":"ContainerStarted","Data":"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0"}
May 25 13:32:26.201120 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:26.201124    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" event={"ID":"ccf79461-bb62-4c85-8266-401065829631","Type":"ContainerStarted","Data":"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1"}
May 25 13:32:26.201725 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:26.201422    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:26.201725 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:26.201571    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:26.202773 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:26.202746    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 25 13:32:26.219776 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:26.219726    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" podStartSLOduration=6.219705617 podStartE2EDuration="6.219705617s" podCreationTimestamp="2026-05-25 13:32:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:32:26.219083791 +0000 UTC m=+1696.257840917" watchObservedRunningTime="2026-05-25 13:32:26.219705617 +0000 UTC m=+1696.258462741"
May 25 13:32:27.204244 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:27.204202    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 25 13:32:32.208890 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:32.208854    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:32.210188 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:32.209488    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 25 13:32:42.210799 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:42.210762    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:50.427210 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.427173    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"]
May 25 13:32:50.427690 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.427550    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kserve-container" containerID="cri-o://92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1" gracePeriod=30
May 25 13:32:50.427690 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.427569    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kube-rbac-proxy" containerID="cri-o://6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0" gracePeriod=30
May 25 13:32:50.517300 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517253    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"]
May 25 13:32:50.517549 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517536    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="storage-initializer"
May 25 13:32:50.517595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517550    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="storage-initializer"
May 25 13:32:50.517595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517565    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="kube-rbac-proxy"
May 25 13:32:50.517595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517572    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="kube-rbac-proxy"
May 25 13:32:50.517595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517581    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="kserve-container"
May 25 13:32:50.517595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517586    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="kserve-container"
May 25 13:32:50.517738 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517634    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="kserve-container"
May 25 13:32:50.517738 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.517642    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e143ee6-df97-41c0-b597-e3638c591e52" containerName="kube-rbac-proxy"
May 25 13:32:50.519650 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.519632    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.521983 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.521964    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 13:32:50.522090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.522021    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
May 25 13:32:50.531869 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.531832    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"]
May 25 13:32:50.601781 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.601728    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.601972 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.601827    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fcf17b-fe9e-486f-b323-b99d07dc2d62-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.601972 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.601855    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrxqg\" (UniqueName: \"kubernetes.io/projected/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kube-api-access-rrxqg\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.601972 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.601887    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fcf17b-fe9e-486f-b323-b99d07dc2d62-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.703432 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.703337    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fcf17b-fe9e-486f-b323-b99d07dc2d62-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.703592 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.703434    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.703592 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.703502    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fcf17b-fe9e-486f-b323-b99d07dc2d62-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.703592 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.703530    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxqg\" (UniqueName: \"kubernetes.io/projected/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kube-api-access-rrxqg\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.703995 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.703969    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.704256 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.704197    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fcf17b-fe9e-486f-b323-b99d07dc2d62-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.705959 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.705936    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fcf17b-fe9e-486f-b323-b99d07dc2d62-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.712365 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.712340    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrxqg\" (UniqueName: \"kubernetes.io/projected/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kube-api-access-rrxqg\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.829466 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.829418    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:50.973520 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:50.973486    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"]
May 25 13:32:51.029360 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:32:51.029323    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fcf17b_fe9e_486f_b323_b99d07dc2d62.slice/crio-bfc57c76f7fcb4f415ae061f75576c4e1433bfa0731b2813e466401832c2acbf WatchSource:0}: Error finding container bfc57c76f7fcb4f415ae061f75576c4e1433bfa0731b2813e466401832c2acbf: Status 404 returned error can't find the container with id bfc57c76f7fcb4f415ae061f75576c4e1433bfa0731b2813e466401832c2acbf
May 25 13:32:51.168263 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.168238    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:51.207245 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.207209    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf79461-bb62-4c85-8266-401065829631-kserve-provision-location\") pod \"ccf79461-bb62-4c85-8266-401065829631\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") "
May 25 13:32:51.207416 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.207288    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf79461-bb62-4c85-8266-401065829631-proxy-tls\") pod \"ccf79461-bb62-4c85-8266-401065829631\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") "
May 25 13:32:51.207416 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.207336    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf79461-bb62-4c85-8266-401065829631-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"ccf79461-bb62-4c85-8266-401065829631\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") "
May 25 13:32:51.207416 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.207377    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgp4v\" (UniqueName: \"kubernetes.io/projected/ccf79461-bb62-4c85-8266-401065829631-kube-api-access-jgp4v\") pod \"ccf79461-bb62-4c85-8266-401065829631\" (UID: \"ccf79461-bb62-4c85-8266-401065829631\") "
May 25 13:32:51.207676 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.207646    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf79461-bb62-4c85-8266-401065829631-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ccf79461-bb62-4c85-8266-401065829631" (UID: "ccf79461-bb62-4c85-8266-401065829631"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:32:51.207732 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.207685    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf79461-bb62-4c85-8266-401065829631-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "ccf79461-bb62-4c85-8266-401065829631" (UID: "ccf79461-bb62-4c85-8266-401065829631"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:32:51.209584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.209561    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf79461-bb62-4c85-8266-401065829631-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ccf79461-bb62-4c85-8266-401065829631" (UID: "ccf79461-bb62-4c85-8266-401065829631"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:32:51.209695 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.209670    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf79461-bb62-4c85-8266-401065829631-kube-api-access-jgp4v" (OuterVolumeSpecName: "kube-api-access-jgp4v") pod "ccf79461-bb62-4c85-8266-401065829631" (UID: "ccf79461-bb62-4c85-8266-401065829631"). InnerVolumeSpecName "kube-api-access-jgp4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:32:51.272946 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.272850    2549 generic.go:358] "Generic (PLEG): container finished" podID="ccf79461-bb62-4c85-8266-401065829631" containerID="6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0" exitCode=2
May 25 13:32:51.272946 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.272878    2549 generic.go:358] "Generic (PLEG): container finished" podID="ccf79461-bb62-4c85-8266-401065829631" containerID="92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1" exitCode=0
May 25 13:32:51.272946 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.272936    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"
May 25 13:32:51.273194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.272929    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" event={"ID":"ccf79461-bb62-4c85-8266-401065829631","Type":"ContainerDied","Data":"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0"}
May 25 13:32:51.273194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.273057    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" event={"ID":"ccf79461-bb62-4c85-8266-401065829631","Type":"ContainerDied","Data":"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1"}
May 25 13:32:51.273194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.273075    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs" event={"ID":"ccf79461-bb62-4c85-8266-401065829631","Type":"ContainerDied","Data":"06bd9582126d18778387608878fff9a490ee1b60bf99fa4bc67474a4de9b5ac8"}
May 25 13:32:51.273194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.273093    2549 scope.go:117] "RemoveContainer" containerID="6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0"
May 25 13:32:51.274469 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.274438    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" event={"ID":"24fcf17b-fe9e-486f-b323-b99d07dc2d62","Type":"ContainerStarted","Data":"4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298"}
May 25 13:32:51.274584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.274474    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" event={"ID":"24fcf17b-fe9e-486f-b323-b99d07dc2d62","Type":"ContainerStarted","Data":"bfc57c76f7fcb4f415ae061f75576c4e1433bfa0731b2813e466401832c2acbf"}
May 25 13:32:51.281368 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.281253    2549 scope.go:117] "RemoveContainer" containerID="92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1"
May 25 13:32:51.288692 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.288668    2549 scope.go:117] "RemoveContainer" containerID="1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace"
May 25 13:32:51.297009 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.296990    2549 scope.go:117] "RemoveContainer" containerID="6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0"
May 25 13:32:51.297311 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:32:51.297285    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0\": container with ID starting with 6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0 not found: ID does not exist" containerID="6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0"
May 25 13:32:51.297397 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.297318    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0"} err="failed to get container status \"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0\": rpc error: code = NotFound desc = could not find container \"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0\": container with ID starting with 6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0 not found: ID does not exist"
May 25 13:32:51.297397 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.297335    2549 scope.go:117] "RemoveContainer" containerID="92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1"
May 25 13:32:51.297625 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:32:51.297606    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1\": container with ID starting with 92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1 not found: ID does not exist" containerID="92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1"
May 25 13:32:51.297670 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.297637    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1"} err="failed to get container status \"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1\": rpc error: code = NotFound desc = could not find container \"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1\": container with ID starting with 92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1 not found: ID does not exist"
May 25 13:32:51.297670 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.297662    2549 scope.go:117] "RemoveContainer" containerID="1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace"
May 25 13:32:51.297928 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:32:51.297910    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace\": container with ID starting with 1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace not found: ID does not exist" containerID="1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace"
May 25 13:32:51.297970 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.297934    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace"} err="failed to get container status \"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace\": rpc error: code = NotFound desc = could not find container \"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace\": container with ID starting with 1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace not found: ID does not exist"
May 25 13:32:51.297970 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.297949    2549 scope.go:117] "RemoveContainer" containerID="6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0"
May 25 13:32:51.298180 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.298163    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0"} err="failed to get container status \"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0\": rpc error: code = NotFound desc = could not find container \"6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0\": container with ID starting with 6ef1b784d3648a243ccc61dfd64f0d87f1504baca9abca6d549cb99e400dcce0 not found: ID does not exist"
May 25 13:32:51.298224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.298180    2549 scope.go:117] "RemoveContainer" containerID="92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1"
May 25 13:32:51.298455 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.298436    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1"} err="failed to get container status \"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1\": rpc error: code = NotFound desc = could not find container \"92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1\": container with ID starting with 92b2a9f349dab2c9c55fb6677e175cf32fd3dc52da4adc49ac11a153b30052b1 not found: ID does not exist"
May 25 13:32:51.298513 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.298455    2549 scope.go:117] "RemoveContainer" containerID="1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace"
May 25 13:32:51.298679 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.298661    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace"} err="failed to get container status \"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace\": rpc error: code = NotFound desc = could not find container \"1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace\": container with ID starting with 1ff5ebd2870c0b2322bc63d6ebeecd9400119446168a23247562073b75dd4ace not found: ID does not exist"
May 25 13:32:51.307588 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.307560    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"]
May 25 13:32:51.307981 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.307953    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccf79461-bb62-4c85-8266-401065829631-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:32:51.308096 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.307987    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgp4v\" (UniqueName: \"kubernetes.io/projected/ccf79461-bb62-4c85-8266-401065829631-kube-api-access-jgp4v\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:32:51.308096 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.308005    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccf79461-bb62-4c85-8266-401065829631-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:32:51.308096 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.308018    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccf79461-bb62-4c85-8266-401065829631-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:32:51.309601 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:51.309579    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-7nmvs"]
May 25 13:32:52.496560 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:52.496526    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccf79461-bb62-4c85-8266-401065829631" path="/var/lib/kubelet/pods/ccf79461-bb62-4c85-8266-401065829631/volumes"
May 25 13:32:55.289520 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:55.289485    2549 generic.go:358] "Generic (PLEG): container finished" podID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerID="4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298" exitCode=0
May 25 13:32:55.289520 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:55.289530    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" event={"ID":"24fcf17b-fe9e-486f-b323-b99d07dc2d62","Type":"ContainerDied","Data":"4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298"}
May 25 13:32:56.294746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:56.294711    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" event={"ID":"24fcf17b-fe9e-486f-b323-b99d07dc2d62","Type":"ContainerStarted","Data":"be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58"}
May 25 13:32:56.294746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:56.294748    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" event={"ID":"24fcf17b-fe9e-486f-b323-b99d07dc2d62","Type":"ContainerStarted","Data":"a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4"}
May 25 13:32:56.295159 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:56.294953    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:56.295159 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:56.294985    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:32:56.312248 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:32:56.312194    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" podStartSLOduration=6.31217794 podStartE2EDuration="6.31217794s" podCreationTimestamp="2026-05-25 13:32:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:32:56.311440046 +0000 UTC m=+1726.350197174" watchObservedRunningTime="2026-05-25 13:32:56.31217794 +0000 UTC m=+1726.350935064"
May 25 13:33:02.307177 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:02.307148    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:33:32.310913 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:32.310881    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:33:40.611193 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.611157    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"]
May 25 13:33:40.611784 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.611500    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="kserve-container" containerID="cri-o://a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4" gracePeriod=30
May 25 13:33:40.611784 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.611535    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="kube-rbac-proxy" containerID="cri-o://be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58" gracePeriod=30
May 25 13:33:40.696887 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.696849    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"]
May 25 13:33:40.697151 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.697139    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="storage-initializer"
May 25 13:33:40.697199 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.697153    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="storage-initializer"
May 25 13:33:40.697199 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.697167    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kserve-container"
May 25 13:33:40.697199 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.697173    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kserve-container"
May 25 13:33:40.697199 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.697180    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kube-rbac-proxy"
May 25 13:33:40.697199 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.697187    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kube-rbac-proxy"
May 25 13:33:40.697375 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.697237    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kube-rbac-proxy"
May 25 13:33:40.697375 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.697247    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ccf79461-bb62-4c85-8266-401065829631" containerName="kserve-container"
May 25 13:33:40.700444 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.700427    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.705090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.705068    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
May 25 13:33:40.705215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.705109    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
May 25 13:33:40.713224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.713199    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"]
May 25 13:33:40.795949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.795910    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8329a0-f355-480e-b3d3-9b67f97917cb-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.795949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.795954    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhflj\" (UniqueName: \"kubernetes.io/projected/bb8329a0-f355-480e-b3d3-9b67f97917cb-kube-api-access-vhflj\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.796196 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.795980    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8329a0-f355-480e-b3d3-9b67f97917cb-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.796196 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.796106    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8329a0-f355-480e-b3d3-9b67f97917cb-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.897289 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.897236    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8329a0-f355-480e-b3d3-9b67f97917cb-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.897469 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.897318    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vhflj\" (UniqueName: \"kubernetes.io/projected/bb8329a0-f355-480e-b3d3-9b67f97917cb-kube-api-access-vhflj\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.897469 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.897351    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8329a0-f355-480e-b3d3-9b67f97917cb-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.897469 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.897397    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8329a0-f355-480e-b3d3-9b67f97917cb-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.897733 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.897711    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8329a0-f355-480e-b3d3-9b67f97917cb-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.898049 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.898025    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8329a0-f355-480e-b3d3-9b67f97917cb-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.899883 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.899859    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8329a0-f355-480e-b3d3-9b67f97917cb-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:40.906193 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:40.906175    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhflj\" (UniqueName: \"kubernetes.io/projected/bb8329a0-f355-480e-b3d3-9b67f97917cb-kube-api-access-vhflj\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:41.012404 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:41.012359    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:41.149056 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:41.148946    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"]
May 25 13:33:41.151697 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:33:41.151671    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8329a0_f355_480e_b3d3_9b67f97917cb.slice/crio-93c851282005f83fb8506f5900993dbf17df5f75371e8a1c6668f60726d871f7 WatchSource:0}: Error finding container 93c851282005f83fb8506f5900993dbf17df5f75371e8a1c6668f60726d871f7: Status 404 returned error can't find the container with id 93c851282005f83fb8506f5900993dbf17df5f75371e8a1c6668f60726d871f7
May 25 13:33:41.419167 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:41.419065    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerStarted","Data":"f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4"}
May 25 13:33:41.419167 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:41.419115    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerStarted","Data":"93c851282005f83fb8506f5900993dbf17df5f75371e8a1c6668f60726d871f7"}
May 25 13:33:41.421011 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:41.420985    2549 generic.go:358] "Generic (PLEG): container finished" podID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerID="be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58" exitCode=2
May 25 13:33:41.421120 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:41.421030    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" event={"ID":"24fcf17b-fe9e-486f-b323-b99d07dc2d62","Type":"ContainerDied","Data":"be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58"}
May 25 13:33:41.851926 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:41.851902    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:33:42.006757 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.006663    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fcf17b-fe9e-486f-b323-b99d07dc2d62-proxy-tls\") pod \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") "
May 25 13:33:42.006757 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.006714    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrxqg\" (UniqueName: \"kubernetes.io/projected/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kube-api-access-rrxqg\") pod \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") "
May 25 13:33:42.006757 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.006757    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fcf17b-fe9e-486f-b323-b99d07dc2d62-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") "
May 25 13:33:42.007051 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.006782    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kserve-provision-location\") pod \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\" (UID: \"24fcf17b-fe9e-486f-b323-b99d07dc2d62\") "
May 25 13:33:42.007149 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.007124    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fcf17b-fe9e-486f-b323-b99d07dc2d62-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "24fcf17b-fe9e-486f-b323-b99d07dc2d62" (UID: "24fcf17b-fe9e-486f-b323-b99d07dc2d62"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:33:42.007229 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.007205    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24fcf17b-fe9e-486f-b323-b99d07dc2d62" (UID: "24fcf17b-fe9e-486f-b323-b99d07dc2d62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:33:42.008986 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.008968    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24fcf17b-fe9e-486f-b323-b99d07dc2d62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24fcf17b-fe9e-486f-b323-b99d07dc2d62" (UID: "24fcf17b-fe9e-486f-b323-b99d07dc2d62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:33:42.009058 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.009030    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kube-api-access-rrxqg" (OuterVolumeSpecName: "kube-api-access-rrxqg") pod "24fcf17b-fe9e-486f-b323-b99d07dc2d62" (UID: "24fcf17b-fe9e-486f-b323-b99d07dc2d62"). InnerVolumeSpecName "kube-api-access-rrxqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:33:42.108223 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.108184    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fcf17b-fe9e-486f-b323-b99d07dc2d62-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:33:42.108223 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.108218    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:33:42.108223 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.108228    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fcf17b-fe9e-486f-b323-b99d07dc2d62-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:33:42.108471 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.108241    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrxqg\" (UniqueName: \"kubernetes.io/projected/24fcf17b-fe9e-486f-b323-b99d07dc2d62-kube-api-access-rrxqg\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:33:42.425779 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.425738    2549 generic.go:358] "Generic (PLEG): container finished" podID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerID="a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4" exitCode=0
May 25 13:33:42.425949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.425813    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" event={"ID":"24fcf17b-fe9e-486f-b323-b99d07dc2d62","Type":"ContainerDied","Data":"a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4"}
May 25 13:33:42.425949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.425834    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"
May 25 13:33:42.425949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.425848    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf" event={"ID":"24fcf17b-fe9e-486f-b323-b99d07dc2d62","Type":"ContainerDied","Data":"bfc57c76f7fcb4f415ae061f75576c4e1433bfa0731b2813e466401832c2acbf"}
May 25 13:33:42.425949 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.425867    2549 scope.go:117] "RemoveContainer" containerID="be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58"
May 25 13:33:42.433995 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.433978    2549 scope.go:117] "RemoveContainer" containerID="a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4"
May 25 13:33:42.440654 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.440638    2549 scope.go:117] "RemoveContainer" containerID="4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298"
May 25 13:33:42.446884 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.446862    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"]
May 25 13:33:42.447659 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.447640    2549 scope.go:117] "RemoveContainer" containerID="be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58"
May 25 13:33:42.447891 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:33:42.447871    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58\": container with ID starting with be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58 not found: ID does not exist" containerID="be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58"
May 25 13:33:42.447942 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.447900    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58"} err="failed to get container status \"be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58\": rpc error: code = NotFound desc = could not find container \"be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58\": container with ID starting with be5277ee95b59a435b0a04e3154aa923a26f627724da54a404bd81d5fae3ac58 not found: ID does not exist"
May 25 13:33:42.447942 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.447921    2549 scope.go:117] "RemoveContainer" containerID="a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4"
May 25 13:33:42.448164 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:33:42.448146    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4\": container with ID starting with a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4 not found: ID does not exist" containerID="a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4"
May 25 13:33:42.448218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.448166    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4"} err="failed to get container status \"a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4\": rpc error: code = NotFound desc = could not find container \"a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4\": container with ID starting with a8cf9083c05ff07843795841da92172946a6a6cf4c6df9dc108ffad889536af4 not found: ID does not exist"
May 25 13:33:42.448218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.448178    2549 scope.go:117] "RemoveContainer" containerID="4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298"
May 25 13:33:42.448435 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:33:42.448419    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298\": container with ID starting with 4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298 not found: ID does not exist" containerID="4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298"
May 25 13:33:42.448470 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.448440    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298"} err="failed to get container status \"4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298\": rpc error: code = NotFound desc = could not find container \"4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298\": container with ID starting with 4fbc89e4ce49cc9b21a9d98b6d06fd4e3a225dbded006baa579af57f11cc4298 not found: ID does not exist"
May 25 13:33:42.450531 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.450512    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-ttddf"]
May 25 13:33:42.495840 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:42.495816    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" path="/var/lib/kubelet/pods/24fcf17b-fe9e-486f-b323-b99d07dc2d62/volumes"
May 25 13:33:45.435003 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:45.434965    2549 generic.go:358] "Generic (PLEG): container finished" podID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerID="f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4" exitCode=0
May 25 13:33:45.435474 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:45.435042    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerDied","Data":"f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4"}
May 25 13:33:46.441034 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:46.440983    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerStarted","Data":"a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff"}
May 25 13:33:48.448854 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:48.448823    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerStarted","Data":"75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4"}
May 25 13:33:48.448854 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:48.448855    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerStarted","Data":"f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e"}
May 25 13:33:48.449243 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:48.448999    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:48.449243 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:48.449057    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:48.475075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:48.474981    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podStartSLOduration=5.754106085 podStartE2EDuration="8.474963109s" podCreationTimestamp="2026-05-25 13:33:40 +0000 UTC" firstStartedPulling="2026-05-25 13:33:45.491655572 +0000 UTC m=+1775.530412676" lastFinishedPulling="2026-05-25 13:33:48.212512598 +0000 UTC m=+1778.251269700" observedRunningTime="2026-05-25 13:33:48.472570596 +0000 UTC m=+1778.511327729" watchObservedRunningTime="2026-05-25 13:33:48.474963109 +0000 UTC m=+1778.513720232"
May 25 13:33:49.452519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:49.452485    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:33:55.460400 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:33:55.460371    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:34:11.101861 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:34:11.101828    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:34:11.104751 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:34:11.104732    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:34:15.461321 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:34:15.461262    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.28:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.28:8080: connect: connection refused"
May 25 13:34:25.463029 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:34:25.462991    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:34:55.463951 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:34:55.463922    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:35:00.766644 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.766609    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"]
May 25 13:35:00.767185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.767034    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-container" containerID="cri-o://a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff" gracePeriod=30
May 25 13:35:00.767185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.767124    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-agent" containerID="cri-o://f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e" gracePeriod=30
May 25 13:35:00.767185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.767135    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy" containerID="cri-o://75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4" gracePeriod=30
May 25 13:35:00.865243 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865205    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"]
May 25 13:35:00.865539 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865525    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="kserve-container"
May 25 13:35:00.865584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865541    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="kserve-container"
May 25 13:35:00.865584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865559    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="kube-rbac-proxy"
May 25 13:35:00.865584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865565    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="kube-rbac-proxy"
May 25 13:35:00.865584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865573    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="storage-initializer"
May 25 13:35:00.865584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865581    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="storage-initializer"
May 25 13:35:00.865734 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865630    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="kube-rbac-proxy"
May 25 13:35:00.865734 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.865642    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="24fcf17b-fe9e-486f-b323-b99d07dc2d62" containerName="kserve-container"
May 25 13:35:00.868584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.868568    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:00.870762 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.870739    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
May 25 13:35:00.870903 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.870805    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
May 25 13:35:00.883955 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.883931    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"]
May 25 13:35:00.996023 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.995969    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52f21a8-9696-4994-9f18-840fa341b37d-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:00.996023 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.996029    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzll\" (UniqueName: \"kubernetes.io/projected/a52f21a8-9696-4994-9f18-840fa341b37d-kube-api-access-9nzll\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:00.996332 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.996060    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52f21a8-9696-4994-9f18-840fa341b37d-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:00.996332 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:00.996080    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52f21a8-9696-4994-9f18-840fa341b37d-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.097090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.096986    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52f21a8-9696-4994-9f18-840fa341b37d-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.097090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.097045    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzll\" (UniqueName: \"kubernetes.io/projected/a52f21a8-9696-4994-9f18-840fa341b37d-kube-api-access-9nzll\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.097090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.097087    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52f21a8-9696-4994-9f18-840fa341b37d-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.097414 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.097114    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52f21a8-9696-4994-9f18-840fa341b37d-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.097544 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.097516    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52f21a8-9696-4994-9f18-840fa341b37d-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.097759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.097735    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52f21a8-9696-4994-9f18-840fa341b37d-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.099828 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.099799    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52f21a8-9696-4994-9f18-840fa341b37d-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.106184 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.106158    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzll\" (UniqueName: \"kubernetes.io/projected/a52f21a8-9696-4994-9f18-840fa341b37d-kube-api-access-9nzll\") pod \"isvc-paddle-predictor-6b8b7cfb4b-5wb54\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.179072 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.179028    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:01.312829 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.312802    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"]
May 25 13:35:01.315480 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:35:01.315449    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52f21a8_9696_4994_9f18_840fa341b37d.slice/crio-7b2e656d277f62a14abe220fa286f9d9bbdc4ec6dad7849397dbf338d2c0d9e0 WatchSource:0}: Error finding container 7b2e656d277f62a14abe220fa286f9d9bbdc4ec6dad7849397dbf338d2c0d9e0: Status 404 returned error can't find the container with id 7b2e656d277f62a14abe220fa286f9d9bbdc4ec6dad7849397dbf338d2c0d9e0
May 25 13:35:01.317166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.317150    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:35:01.646851 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.646806    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" event={"ID":"a52f21a8-9696-4994-9f18-840fa341b37d","Type":"ContainerStarted","Data":"f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562"}
May 25 13:35:01.647065 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.646857    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" event={"ID":"a52f21a8-9696-4994-9f18-840fa341b37d","Type":"ContainerStarted","Data":"7b2e656d277f62a14abe220fa286f9d9bbdc4ec6dad7849397dbf338d2c0d9e0"}
May 25 13:35:01.649193 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.649162    2549 generic.go:358] "Generic (PLEG): container finished" podID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerID="75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4" exitCode=2
May 25 13:35:01.649351 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:01.649246    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerDied","Data":"75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4"}
May 25 13:35:03.655733 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:03.655703    2549 generic.go:358] "Generic (PLEG): container finished" podID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerID="a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff" exitCode=0
May 25 13:35:03.656097 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:03.655786    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerDied","Data":"a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff"}
May 25 13:35:05.455340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:05.455294    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:35:05.461579 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:05.461546    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.28:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.28:8080: connect: connection refused"
May 25 13:35:06.666046 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:06.666010    2549 generic.go:358] "Generic (PLEG): container finished" podID="a52f21a8-9696-4994-9f18-840fa341b37d" containerID="f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562" exitCode=0
May 25 13:35:06.666510 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:06.666088    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" event={"ID":"a52f21a8-9696-4994-9f18-840fa341b37d","Type":"ContainerDied","Data":"f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562"}
May 25 13:35:10.456221 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:10.456166    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:35:15.455982 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:15.455925    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:35:15.456497 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:15.456112    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:35:15.461426 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:15.461388    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.28:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.28:8080: connect: connection refused"
May 25 13:35:18.708034 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:18.707937    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" event={"ID":"a52f21a8-9696-4994-9f18-840fa341b37d","Type":"ContainerStarted","Data":"ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae"}
May 25 13:35:18.708034 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:18.707982    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" event={"ID":"a52f21a8-9696-4994-9f18-840fa341b37d","Type":"ContainerStarted","Data":"3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2"}
May 25 13:35:18.708624 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:18.708253    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:18.708624 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:18.708381    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:18.709551 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:18.709524    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:35:18.728306 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:18.728243    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podStartSLOduration=7.6899341450000005 podStartE2EDuration="18.728231207s" podCreationTimestamp="2026-05-25 13:35:00 +0000 UTC" firstStartedPulling="2026-05-25 13:35:06.667366115 +0000 UTC m=+1856.706123218" lastFinishedPulling="2026-05-25 13:35:17.705663164 +0000 UTC m=+1867.744420280" observedRunningTime="2026-05-25 13:35:18.727046186 +0000 UTC m=+1868.765803324" watchObservedRunningTime="2026-05-25 13:35:18.728231207 +0000 UTC m=+1868.766988383"
May 25 13:35:19.711459 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:19.711425    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:35:20.455747 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:20.455705    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:35:24.715985 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:24.715953    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:35:24.716447 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:24.716404    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:35:25.455406 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:25.455356    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:35:25.461806 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:25.461775    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.28:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.28:8080: connect: connection refused"
May 25 13:35:25.461945 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:25.461903    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:35:30.456202 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:30.456156    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 25 13:35:30.951984 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:30.951960    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:35:31.036557 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.036467    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8329a0-f355-480e-b3d3-9b67f97917cb-proxy-tls\") pod \"bb8329a0-f355-480e-b3d3-9b67f97917cb\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") "
May 25 13:35:31.036557 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.036508    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8329a0-f355-480e-b3d3-9b67f97917cb-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"bb8329a0-f355-480e-b3d3-9b67f97917cb\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") "
May 25 13:35:31.036557 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.036528    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhflj\" (UniqueName: \"kubernetes.io/projected/bb8329a0-f355-480e-b3d3-9b67f97917cb-kube-api-access-vhflj\") pod \"bb8329a0-f355-480e-b3d3-9b67f97917cb\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") "
May 25 13:35:31.036765 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.036572    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8329a0-f355-480e-b3d3-9b67f97917cb-kserve-provision-location\") pod \"bb8329a0-f355-480e-b3d3-9b67f97917cb\" (UID: \"bb8329a0-f355-480e-b3d3-9b67f97917cb\") "
May 25 13:35:31.036882 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.036847    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8329a0-f355-480e-b3d3-9b67f97917cb-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "bb8329a0-f355-480e-b3d3-9b67f97917cb" (UID: "bb8329a0-f355-480e-b3d3-9b67f97917cb"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:35:31.037012 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.036906    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8329a0-f355-480e-b3d3-9b67f97917cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb8329a0-f355-480e-b3d3-9b67f97917cb" (UID: "bb8329a0-f355-480e-b3d3-9b67f97917cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:35:31.038746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.038720    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8329a0-f355-480e-b3d3-9b67f97917cb-kube-api-access-vhflj" (OuterVolumeSpecName: "kube-api-access-vhflj") pod "bb8329a0-f355-480e-b3d3-9b67f97917cb" (UID: "bb8329a0-f355-480e-b3d3-9b67f97917cb"). InnerVolumeSpecName "kube-api-access-vhflj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:35:31.038838 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.038774    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8329a0-f355-480e-b3d3-9b67f97917cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb8329a0-f355-480e-b3d3-9b67f97917cb" (UID: "bb8329a0-f355-480e-b3d3-9b67f97917cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:35:31.137502 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.137464    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8329a0-f355-480e-b3d3-9b67f97917cb-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:35:31.137502 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.137495    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8329a0-f355-480e-b3d3-9b67f97917cb-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:35:31.137502 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.137506    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhflj\" (UniqueName: \"kubernetes.io/projected/bb8329a0-f355-480e-b3d3-9b67f97917cb-kube-api-access-vhflj\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:35:31.137502 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.137516    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8329a0-f355-480e-b3d3-9b67f97917cb-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:35:31.746619 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.746587    2549 generic.go:358] "Generic (PLEG): container finished" podID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerID="f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e" exitCode=0
May 25 13:35:31.747045 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.746662    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerDied","Data":"f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e"}
May 25 13:35:31.747045 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.746684    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"
May 25 13:35:31.747045 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.746702    2549 scope.go:117] "RemoveContainer" containerID="75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4"
May 25 13:35:31.747045 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.746692    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc" event={"ID":"bb8329a0-f355-480e-b3d3-9b67f97917cb","Type":"ContainerDied","Data":"93c851282005f83fb8506f5900993dbf17df5f75371e8a1c6668f60726d871f7"}
May 25 13:35:31.754328 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.754309    2549 scope.go:117] "RemoveContainer" containerID="f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e"
May 25 13:35:31.761850 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.761830    2549 scope.go:117] "RemoveContainer" containerID="a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff"
May 25 13:35:31.768800 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.768776    2549 scope.go:117] "RemoveContainer" containerID="f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4"
May 25 13:35:31.770109 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.770089    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"]
May 25 13:35:31.774876 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.774849    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-hlgwc"]
May 25 13:35:31.777361 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.777203    2549 scope.go:117] "RemoveContainer" containerID="75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4"
May 25 13:35:31.777668 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:35:31.777647    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4\": container with ID starting with 75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4 not found: ID does not exist" containerID="75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4"
May 25 13:35:31.777718 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.777679    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4"} err="failed to get container status \"75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4\": rpc error: code = NotFound desc = could not find container \"75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4\": container with ID starting with 75b0be58350ebcee8f8a6945d3232c41ac2642afffb0247b37bf875673bd58b4 not found: ID does not exist"
May 25 13:35:31.777718 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.777699    2549 scope.go:117] "RemoveContainer" containerID="f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e"
May 25 13:35:31.777934 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:35:31.777915    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e\": container with ID starting with f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e not found: ID does not exist" containerID="f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e"
May 25 13:35:31.777979 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.777941    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e"} err="failed to get container status \"f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e\": rpc error: code = NotFound desc = could not find container \"f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e\": container with ID starting with f150de719733e9c305a7c5e036ff5bcc5369214b90c9c9dd5f75ea6721a1500e not found: ID does not exist"
May 25 13:35:31.777979 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.777958    2549 scope.go:117] "RemoveContainer" containerID="a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff"
May 25 13:35:31.778183 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:35:31.778166    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff\": container with ID starting with a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff not found: ID does not exist" containerID="a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff"
May 25 13:35:31.778244 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.778191    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff"} err="failed to get container status \"a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff\": rpc error: code = NotFound desc = could not find container \"a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff\": container with ID starting with a183e3f50744cf1be961639ce76ccf96cf0da7c35ffd7dbee936585c98a147ff not found: ID does not exist"
May 25 13:35:31.778244 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.778214    2549 scope.go:117] "RemoveContainer" containerID="f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4"
May 25 13:35:31.778542 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:35:31.778526    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4\": container with ID starting with f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4 not found: ID does not exist" containerID="f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4"
May 25 13:35:31.778596 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:31.778546    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4"} err="failed to get container status \"f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4\": rpc error: code = NotFound desc = could not find container \"f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4\": container with ID starting with f9171859c23dc0847d453b220c14a566c716affb7b6ee39c863ad77434cdcbc4 not found: ID does not exist"
May 25 13:35:32.496849 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:32.496815    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" path="/var/lib/kubelet/pods/bb8329a0-f355-480e-b3d3-9b67f97917cb/volumes"
May 25 13:35:34.716785 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:34.716733    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:35:44.717128 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:44.717081    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:35:54.716495 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:35:54.716454    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:36:04.717438 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:04.717403    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:36:12.263942 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.263905    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"]
May 25 13:36:12.264383 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.264206    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container" containerID="cri-o://3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2" gracePeriod=30
May 25 13:36:12.264383 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.264261    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kube-rbac-proxy" containerID="cri-o://ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae" gracePeriod=30
May 25 13:36:12.359612 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359568    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"]
May 25 13:36:12.359897 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359881    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-container"
May 25 13:36:12.359965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359901    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-container"
May 25 13:36:12.359965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359911    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-agent"
May 25 13:36:12.359965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359916    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-agent"
May 25 13:36:12.359965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359925    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="storage-initializer"
May 25 13:36:12.359965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359930    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="storage-initializer"
May 25 13:36:12.359965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359940    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy"
May 25 13:36:12.359965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359945    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy"
May 25 13:36:12.360189 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.359999    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-agent"
May 25 13:36:12.360189 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.360009    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kube-rbac-proxy"
May 25 13:36:12.360189 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.360017    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb8329a0-f355-480e-b3d3-9b67f97917cb" containerName="kserve-container"
May 25 13:36:12.364044 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.364027    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.366811 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.366787    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
May 25 13:36:12.366939 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.366858    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
May 25 13:36:12.376718 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.376690    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"]
May 25 13:36:12.460024 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.459988    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.460024 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.460033    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.460258 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.460059    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.460258 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.460113    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlg8w\" (UniqueName: \"kubernetes.io/projected/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kube-api-access-wlg8w\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.560841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.560739    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.560841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.560796    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.560841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.560827    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.561111 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.560863    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wlg8w\" (UniqueName: \"kubernetes.io/projected/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kube-api-access-wlg8w\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.561111 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:36:12.560904    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
May 25 13:36:12.561111 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:36:12.560978    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls podName:4ad4a5da-14cd-4f79-bc2f-b44f73af5457 nodeName:}" failed. No retries permitted until 2026-05-25 13:36:13.060956004 +0000 UTC m=+1923.099713106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls") pod "isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" (UID: "4ad4a5da-14cd-4f79-bc2f-b44f73af5457") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
May 25 13:36:12.561324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.561303    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.561580 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.561561    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.573185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.573159    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlg8w\" (UniqueName: \"kubernetes.io/projected/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kube-api-access-wlg8w\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:12.863680 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.863588    2549 generic.go:358] "Generic (PLEG): container finished" podID="a52f21a8-9696-4994-9f18-840fa341b37d" containerID="ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae" exitCode=2
May 25 13:36:12.863822 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:12.863670    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" event={"ID":"a52f21a8-9696-4994-9f18-840fa341b37d","Type":"ContainerDied","Data":"ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae"}
May 25 13:36:13.063825 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:13.063793    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:13.066280 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:13.066245    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:13.275819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:13.275778    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:13.397393 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:13.397352    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"]
May 25 13:36:13.401208 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:36:13.401180    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad4a5da_14cd_4f79_bc2f_b44f73af5457.slice/crio-c103acfd28c5f1a30287b5f16b80aafc4a521d8dd64cee86901409011c47c996 WatchSource:0}: Error finding container c103acfd28c5f1a30287b5f16b80aafc4a521d8dd64cee86901409011c47c996: Status 404 returned error can't find the container with id c103acfd28c5f1a30287b5f16b80aafc4a521d8dd64cee86901409011c47c996
May 25 13:36:13.867803 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:13.867760    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" event={"ID":"4ad4a5da-14cd-4f79-bc2f-b44f73af5457","Type":"ContainerStarted","Data":"5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c"}
May 25 13:36:13.867803 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:13.867805    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" event={"ID":"4ad4a5da-14cd-4f79-bc2f-b44f73af5457","Type":"ContainerStarted","Data":"c103acfd28c5f1a30287b5f16b80aafc4a521d8dd64cee86901409011c47c996"}
May 25 13:36:14.712328 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:14.712255    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
May 25 13:36:14.716547 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:14.716523    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 25 13:36:15.008654 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.008630    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:36:15.081910 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.081879    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nzll\" (UniqueName: \"kubernetes.io/projected/a52f21a8-9696-4994-9f18-840fa341b37d-kube-api-access-9nzll\") pod \"a52f21a8-9696-4994-9f18-840fa341b37d\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") "
May 25 13:36:15.082075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.081951    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52f21a8-9696-4994-9f18-840fa341b37d-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"a52f21a8-9696-4994-9f18-840fa341b37d\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") "
May 25 13:36:15.082075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.081969    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52f21a8-9696-4994-9f18-840fa341b37d-kserve-provision-location\") pod \"a52f21a8-9696-4994-9f18-840fa341b37d\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") "
May 25 13:36:15.082075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.082000    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52f21a8-9696-4994-9f18-840fa341b37d-proxy-tls\") pod \"a52f21a8-9696-4994-9f18-840fa341b37d\" (UID: \"a52f21a8-9696-4994-9f18-840fa341b37d\") "
May 25 13:36:15.082359 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.082335    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52f21a8-9696-4994-9f18-840fa341b37d-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "a52f21a8-9696-4994-9f18-840fa341b37d" (UID: "a52f21a8-9696-4994-9f18-840fa341b37d"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:36:15.084064 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.084039    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52f21a8-9696-4994-9f18-840fa341b37d-kube-api-access-9nzll" (OuterVolumeSpecName: "kube-api-access-9nzll") pod "a52f21a8-9696-4994-9f18-840fa341b37d" (UID: "a52f21a8-9696-4994-9f18-840fa341b37d"). InnerVolumeSpecName "kube-api-access-9nzll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:36:15.084161 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.084080    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52f21a8-9696-4994-9f18-840fa341b37d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a52f21a8-9696-4994-9f18-840fa341b37d" (UID: "a52f21a8-9696-4994-9f18-840fa341b37d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:15.097426 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.097393    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f21a8-9696-4994-9f18-840fa341b37d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a52f21a8-9696-4994-9f18-840fa341b37d" (UID: "a52f21a8-9696-4994-9f18-840fa341b37d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:36:15.182750 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.182713    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzll\" (UniqueName: \"kubernetes.io/projected/a52f21a8-9696-4994-9f18-840fa341b37d-kube-api-access-9nzll\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:36:15.182750 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.182743    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a52f21a8-9696-4994-9f18-840fa341b37d-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:36:15.182750 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.182754    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a52f21a8-9696-4994-9f18-840fa341b37d-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:36:15.182968 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.182763    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a52f21a8-9696-4994-9f18-840fa341b37d-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:36:15.874445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.874406    2549 generic.go:358] "Generic (PLEG): container finished" podID="a52f21a8-9696-4994-9f18-840fa341b37d" containerID="3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2" exitCode=0
May 25 13:36:15.874925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.874451    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" event={"ID":"a52f21a8-9696-4994-9f18-840fa341b37d","Type":"ContainerDied","Data":"3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2"}
May 25 13:36:15.874925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.874480    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54" event={"ID":"a52f21a8-9696-4994-9f18-840fa341b37d","Type":"ContainerDied","Data":"7b2e656d277f62a14abe220fa286f9d9bbdc4ec6dad7849397dbf338d2c0d9e0"}
May 25 13:36:15.874925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.874496    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"
May 25 13:36:15.874925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.874499    2549 scope.go:117] "RemoveContainer" containerID="ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae"
May 25 13:36:15.882547 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.882423    2549 scope.go:117] "RemoveContainer" containerID="3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2"
May 25 13:36:15.891625 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.891603    2549 scope.go:117] "RemoveContainer" containerID="f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562"
May 25 13:36:15.896671 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.896644    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"]
May 25 13:36:15.899123 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.899098    2549 scope.go:117] "RemoveContainer" containerID="ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae"
May 25 13:36:15.899429 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:36:15.899401    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae\": container with ID starting with ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae not found: ID does not exist" containerID="ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae"
May 25 13:36:15.899516 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.899437    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae"} err="failed to get container status \"ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae\": rpc error: code = NotFound desc = could not find container \"ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae\": container with ID starting with ed35d9aa8eb538474ba371ce64104535e4d2104976452933a9fc2499044982ae not found: ID does not exist"
May 25 13:36:15.899516 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.899458    2549 scope.go:117] "RemoveContainer" containerID="3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2"
May 25 13:36:15.899717 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:36:15.899703    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2\": container with ID starting with 3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2 not found: ID does not exist" containerID="3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2"
May 25 13:36:15.899758 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.899734    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2"} err="failed to get container status \"3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2\": rpc error: code = NotFound desc = could not find container \"3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2\": container with ID starting with 3436626b0c070b5989d0c17d62bca27820a0f96f49ef2f6dfd3605162fd590a2 not found: ID does not exist"
May 25 13:36:15.899758 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.899752    2549 scope.go:117] "RemoveContainer" containerID="f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562"
May 25 13:36:15.899987 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:36:15.899963    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562\": container with ID starting with f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562 not found: ID does not exist" containerID="f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562"
May 25 13:36:15.900043 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.899991    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562"} err="failed to get container status \"f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562\": rpc error: code = NotFound desc = could not find container \"f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562\": container with ID starting with f74c0ba1db07a07073539a976759571437672dcb7bd49193dcc3ed0e68ac1562 not found: ID does not exist"
May 25 13:36:15.900445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:15.900425    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-5wb54"]
May 25 13:36:16.496072 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:16.496036    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" path="/var/lib/kubelet/pods/a52f21a8-9696-4994-9f18-840fa341b37d/volumes"
May 25 13:36:17.882307 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:17.882253    2549 generic.go:358] "Generic (PLEG): container finished" podID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerID="5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c" exitCode=0
May 25 13:36:17.882664 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:17.882317    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" event={"ID":"4ad4a5da-14cd-4f79-bc2f-b44f73af5457","Type":"ContainerDied","Data":"5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c"}
May 25 13:36:18.887245 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:18.887208    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" event={"ID":"4ad4a5da-14cd-4f79-bc2f-b44f73af5457","Type":"ContainerStarted","Data":"d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd"}
May 25 13:36:18.887245 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:18.887249    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" event={"ID":"4ad4a5da-14cd-4f79-bc2f-b44f73af5457","Type":"ContainerStarted","Data":"bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92"}
May 25 13:36:18.887775 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:18.887546    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:18.887775 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:18.887644    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:18.888953 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:18.888925    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:36:18.907836 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:18.907781    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podStartSLOduration=6.9077680390000005 podStartE2EDuration="6.907768039s" podCreationTimestamp="2026-05-25 13:36:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:36:18.90621587 +0000 UTC m=+1928.944973004" watchObservedRunningTime="2026-05-25 13:36:18.907768039 +0000 UTC m=+1928.946525162"
May 25 13:36:19.890829 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:19.890788    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:36:24.895474 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:24.895445    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:36:24.895989 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:24.895959    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:36:34.896236 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:34.896193    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:36:44.896475 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:44.896428    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:36:54.896841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:36:54.896757    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:37:04.896484 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:04.896452    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:37:13.836981 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.836948    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"]
May 25 13:37:13.837382 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.837239    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container" containerID="cri-o://bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92" gracePeriod=30
May 25 13:37:13.837382 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.837317    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kube-rbac-proxy" containerID="cri-o://d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd" gracePeriod=30
May 25 13:37:13.921618 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921581    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"]
May 25 13:37:13.921865 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921852    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kube-rbac-proxy"
May 25 13:37:13.921924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921867    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kube-rbac-proxy"
May 25 13:37:13.921924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921879    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container"
May 25 13:37:13.921924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921884    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container"
May 25 13:37:13.921924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921891    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="storage-initializer"
May 25 13:37:13.921924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921897    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="storage-initializer"
May 25 13:37:13.922090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921964    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kube-rbac-proxy"
May 25 13:37:13.922090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.921972    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="a52f21a8-9696-4994-9f18-840fa341b37d" containerName="kserve-container"
May 25 13:37:13.925868 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.925852    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:13.927770 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.927741    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 13:37:13.927770 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.927762    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
May 25 13:37:13.937981 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.936383    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"]
May 25 13:37:13.988753 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.988719    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971e7680-ff0f-4010-9f41-de435da8b942-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:13.988888 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.988759    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpdg\" (UniqueName: \"kubernetes.io/projected/971e7680-ff0f-4010-9f41-de435da8b942-kube-api-access-5cpdg\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:13.988888 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.988795    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971e7680-ff0f-4010-9f41-de435da8b942-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:13.988888 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:13.988828    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971e7680-ff0f-4010-9f41-de435da8b942-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.037395 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.037364    2549 generic.go:358] "Generic (PLEG): container finished" podID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerID="d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd" exitCode=2
May 25 13:37:14.037534 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.037420    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" event={"ID":"4ad4a5da-14cd-4f79-bc2f-b44f73af5457","Type":"ContainerDied","Data":"d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd"}
May 25 13:37:14.090008 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.089938    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971e7680-ff0f-4010-9f41-de435da8b942-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.090008 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.089980    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971e7680-ff0f-4010-9f41-de435da8b942-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.090188 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.090049    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971e7680-ff0f-4010-9f41-de435da8b942-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.090188 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.090086    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpdg\" (UniqueName: \"kubernetes.io/projected/971e7680-ff0f-4010-9f41-de435da8b942-kube-api-access-5cpdg\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.090509 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.090479    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971e7680-ff0f-4010-9f41-de435da8b942-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.090793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.090770    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971e7680-ff0f-4010-9f41-de435da8b942-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.092546 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.092528    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971e7680-ff0f-4010-9f41-de435da8b942-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.098434 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.098417    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpdg\" (UniqueName: \"kubernetes.io/projected/971e7680-ff0f-4010-9f41-de435da8b942-kube-api-access-5cpdg\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.241317 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.241258    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:14.369179 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.369149    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"]
May 25 13:37:14.372266 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:37:14.372241    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971e7680_ff0f_4010_9f41_de435da8b942.slice/crio-46cfcf9119cdd8682a2cdf63a691468f1c7b5ce97ec038dad52b35d5386f1121 WatchSource:0}: Error finding container 46cfcf9119cdd8682a2cdf63a691468f1c7b5ce97ec038dad52b35d5386f1121: Status 404 returned error can't find the container with id 46cfcf9119cdd8682a2cdf63a691468f1c7b5ce97ec038dad52b35d5386f1121
May 25 13:37:14.891515 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.891456    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
May 25 13:37:14.896587 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:14.896552    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
May 25 13:37:15.041704 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:15.041647    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" event={"ID":"971e7680-ff0f-4010-9f41-de435da8b942","Type":"ContainerStarted","Data":"92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae"}
May 25 13:37:15.041704 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:15.041703    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" event={"ID":"971e7680-ff0f-4010-9f41-de435da8b942","Type":"ContainerStarted","Data":"46cfcf9119cdd8682a2cdf63a691468f1c7b5ce97ec038dad52b35d5386f1121"}
May 25 13:37:16.576734 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.576710    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:37:16.608374 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.608347    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") "
May 25 13:37:16.608542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.608391    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kserve-provision-location\") pod \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") "
May 25 13:37:16.608542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.608418    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlg8w\" (UniqueName: \"kubernetes.io/projected/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kube-api-access-wlg8w\") pod \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") "
May 25 13:37:16.608542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.608481    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls\") pod \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\" (UID: \"4ad4a5da-14cd-4f79-bc2f-b44f73af5457\") "
May 25 13:37:16.608736 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.608713    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "4ad4a5da-14cd-4f79-bc2f-b44f73af5457" (UID: "4ad4a5da-14cd-4f79-bc2f-b44f73af5457"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:37:16.611194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.611164    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ad4a5da-14cd-4f79-bc2f-b44f73af5457" (UID: "4ad4a5da-14cd-4f79-bc2f-b44f73af5457"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:37:16.611325 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.611187    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kube-api-access-wlg8w" (OuterVolumeSpecName: "kube-api-access-wlg8w") pod "4ad4a5da-14cd-4f79-bc2f-b44f73af5457" (UID: "4ad4a5da-14cd-4f79-bc2f-b44f73af5457"). InnerVolumeSpecName "kube-api-access-wlg8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:37:16.621804 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.621771    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ad4a5da-14cd-4f79-bc2f-b44f73af5457" (UID: "4ad4a5da-14cd-4f79-bc2f-b44f73af5457"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:37:16.709550 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.709461    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:37:16.709550 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.709491    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:37:16.709550 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.709502    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlg8w\" (UniqueName: \"kubernetes.io/projected/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-kube-api-access-wlg8w\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:37:16.709550 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:16.709511    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad4a5da-14cd-4f79-bc2f-b44f73af5457-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:37:17.048433 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.048339    2549 generic.go:358] "Generic (PLEG): container finished" podID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerID="bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92" exitCode=0
May 25 13:37:17.048433 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.048383    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" event={"ID":"4ad4a5da-14cd-4f79-bc2f-b44f73af5457","Type":"ContainerDied","Data":"bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92"}
May 25 13:37:17.048433 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.048411    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp" event={"ID":"4ad4a5da-14cd-4f79-bc2f-b44f73af5457","Type":"ContainerDied","Data":"c103acfd28c5f1a30287b5f16b80aafc4a521d8dd64cee86901409011c47c996"}
May 25 13:37:17.048433 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.048421    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"
May 25 13:37:17.048433 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.048430    2549 scope.go:117] "RemoveContainer" containerID="d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd"
May 25 13:37:17.056064 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.056021    2549 scope.go:117] "RemoveContainer" containerID="bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92"
May 25 13:37:17.063176 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.063159    2549 scope.go:117] "RemoveContainer" containerID="5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c"
May 25 13:37:17.069734 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.069711    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"]
May 25 13:37:17.070325 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.070306    2549 scope.go:117] "RemoveContainer" containerID="d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd"
May 25 13:37:17.070588 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:37:17.070568    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd\": container with ID starting with d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd not found: ID does not exist" containerID="d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd"
May 25 13:37:17.070663 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.070595    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd"} err="failed to get container status \"d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd\": rpc error: code = NotFound desc = could not find container \"d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd\": container with ID starting with d084b5e7f964e5e76584328a14f8e6b74eb432c6fd9b4478b2311b17885167fd not found: ID does not exist"
May 25 13:37:17.070663 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.070612    2549 scope.go:117] "RemoveContainer" containerID="bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92"
May 25 13:37:17.070866 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:37:17.070852    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92\": container with ID starting with bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92 not found: ID does not exist" containerID="bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92"
May 25 13:37:17.070908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.070868    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92"} err="failed to get container status \"bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92\": rpc error: code = NotFound desc = could not find container \"bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92\": container with ID starting with bde718e4bd2fa503d33c9ba938776c8b5a07d7c3cba0f9b9751e9d304ef67b92 not found: ID does not exist"
May 25 13:37:17.070908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.070881    2549 scope.go:117] "RemoveContainer" containerID="5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c"
May 25 13:37:17.071115 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:37:17.071095    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c\": container with ID starting with 5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c not found: ID does not exist" containerID="5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c"
May 25 13:37:17.071152 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.071123    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c"} err="failed to get container status \"5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c\": rpc error: code = NotFound desc = could not find container \"5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c\": container with ID starting with 5de335707d04787232d45387bce756c2dd5bed44185e0b848f170000141d6d8c not found: ID does not exist"
May 25 13:37:17.075392 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:17.075368    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-nxhnp"]
May 25 13:37:18.497046 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:18.497007    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" path="/var/lib/kubelet/pods/4ad4a5da-14cd-4f79-bc2f-b44f73af5457/volumes"
May 25 13:37:19.056115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:19.056083    2549 generic.go:358] "Generic (PLEG): container finished" podID="971e7680-ff0f-4010-9f41-de435da8b942" containerID="92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae" exitCode=0
May 25 13:37:19.056285 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:19.056143    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" event={"ID":"971e7680-ff0f-4010-9f41-de435da8b942","Type":"ContainerDied","Data":"92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae"}
May 25 13:37:20.060858 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:20.060820    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" event={"ID":"971e7680-ff0f-4010-9f41-de435da8b942","Type":"ContainerStarted","Data":"4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48"}
May 25 13:37:20.060858 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:20.060858    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" event={"ID":"971e7680-ff0f-4010-9f41-de435da8b942","Type":"ContainerStarted","Data":"dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3"}
May 25 13:37:20.061259 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:20.061148    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:20.079779 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:20.079726    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podStartSLOduration=7.079710257 podStartE2EDuration="7.079710257s" podCreationTimestamp="2026-05-25 13:37:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:37:20.07895928 +0000 UTC m=+1990.117716403" watchObservedRunningTime="2026-05-25 13:37:20.079710257 +0000 UTC m=+1990.118467384"
May 25 13:37:21.064664 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:21.064624    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:21.065835 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:21.065809    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:37:22.067319 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:22.067283    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:37:27.071948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:27.071920    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:37:27.072523 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:27.072472    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:37:37.073313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:37.073251    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:37:47.072643 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:47.072601    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:37:57.073070 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:37:57.073027    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:38:07.073426 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:07.073391    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:38:15.652193 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.652160    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"]
May 25 13:38:15.652647 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.652495    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container" containerID="cri-o://dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3" gracePeriod=30
May 25 13:38:15.652647 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.652540    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kube-rbac-proxy" containerID="cri-o://4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48" gracePeriod=30
May 25 13:38:15.740518 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740475    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"]
May 25 13:38:15.740776 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740762    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kube-rbac-proxy"
May 25 13:38:15.740819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740778    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kube-rbac-proxy"
May 25 13:38:15.740819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740794    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container"
May 25 13:38:15.740819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740800    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container"
May 25 13:38:15.740819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740808    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="storage-initializer"
May 25 13:38:15.740819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740814    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="storage-initializer"
May 25 13:38:15.740967 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740862    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kserve-container"
May 25 13:38:15.740967 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.740869    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ad4a5da-14cd-4f79-bc2f-b44f73af5457" containerName="kube-rbac-proxy"
May 25 13:38:15.744957 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.744942    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.747194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.747159    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
May 25 13:38:15.747393 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.747374    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
May 25 13:38:15.752158 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.752133    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"]
May 25 13:38:15.848351 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.848308    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47039c40-ae42-4a88-97a7-9666f15db2ff-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.848351 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.848359    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47039c40-ae42-4a88-97a7-9666f15db2ff-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.848565 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.848391    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvflj\" (UniqueName: \"kubernetes.io/projected/47039c40-ae42-4a88-97a7-9666f15db2ff-kube-api-access-xvflj\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.848565 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.848422    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47039c40-ae42-4a88-97a7-9666f15db2ff-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.949015 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.948920    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47039c40-ae42-4a88-97a7-9666f15db2ff-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.949015 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.948972    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xvflj\" (UniqueName: \"kubernetes.io/projected/47039c40-ae42-4a88-97a7-9666f15db2ff-kube-api-access-xvflj\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.949015 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.948999    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47039c40-ae42-4a88-97a7-9666f15db2ff-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.949252 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.949033    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47039c40-ae42-4a88-97a7-9666f15db2ff-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.949542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.949518    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47039c40-ae42-4a88-97a7-9666f15db2ff-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.949852 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.949784    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47039c40-ae42-4a88-97a7-9666f15db2ff-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.951648 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.951623    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47039c40-ae42-4a88-97a7-9666f15db2ff-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:15.956535 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:15.956516    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvflj\" (UniqueName: \"kubernetes.io/projected/47039c40-ae42-4a88-97a7-9666f15db2ff-kube-api-access-xvflj\") pod \"isvc-pmml-predictor-8bb578669-rtct6\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:16.055370 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:16.055328    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:16.186438 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:16.186408    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"]
May 25 13:38:16.188723 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:38:16.188696    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47039c40_ae42_4a88_97a7_9666f15db2ff.slice/crio-f1d323947da20de810768d3d4e622462e8405d9872355abd3f56ec186c2cd8ab WatchSource:0}: Error finding container f1d323947da20de810768d3d4e622462e8405d9872355abd3f56ec186c2cd8ab: Status 404 returned error can't find the container with id f1d323947da20de810768d3d4e622462e8405d9872355abd3f56ec186c2cd8ab
May 25 13:38:16.216730 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:16.216702    2549 generic.go:358] "Generic (PLEG): container finished" podID="971e7680-ff0f-4010-9f41-de435da8b942" containerID="4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48" exitCode=2
May 25 13:38:16.216844 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:16.216772    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" event={"ID":"971e7680-ff0f-4010-9f41-de435da8b942","Type":"ContainerDied","Data":"4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48"}
May 25 13:38:16.217772 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:16.217740    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" event={"ID":"47039c40-ae42-4a88-97a7-9666f15db2ff","Type":"ContainerStarted","Data":"f1d323947da20de810768d3d4e622462e8405d9872355abd3f56ec186c2cd8ab"}
May 25 13:38:17.068401 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:17.068352    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
May 25 13:38:17.072663 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:17.072636    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 25 13:38:17.221626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:17.221587    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" event={"ID":"47039c40-ae42-4a88-97a7-9666f15db2ff","Type":"ContainerStarted","Data":"cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d"}
May 25 13:38:18.392789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.392766    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:38:18.568616 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.568521    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971e7680-ff0f-4010-9f41-de435da8b942-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"971e7680-ff0f-4010-9f41-de435da8b942\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") "
May 25 13:38:18.568616 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.568580    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cpdg\" (UniqueName: \"kubernetes.io/projected/971e7680-ff0f-4010-9f41-de435da8b942-kube-api-access-5cpdg\") pod \"971e7680-ff0f-4010-9f41-de435da8b942\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") "
May 25 13:38:18.568616 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.568598    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971e7680-ff0f-4010-9f41-de435da8b942-proxy-tls\") pod \"971e7680-ff0f-4010-9f41-de435da8b942\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") "
May 25 13:38:18.568852 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.568652    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971e7680-ff0f-4010-9f41-de435da8b942-kserve-provision-location\") pod \"971e7680-ff0f-4010-9f41-de435da8b942\" (UID: \"971e7680-ff0f-4010-9f41-de435da8b942\") "
May 25 13:38:18.569045 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.569009    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971e7680-ff0f-4010-9f41-de435da8b942-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "971e7680-ff0f-4010-9f41-de435da8b942" (UID: "971e7680-ff0f-4010-9f41-de435da8b942"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:38:18.570936 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.570890    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971e7680-ff0f-4010-9f41-de435da8b942-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "971e7680-ff0f-4010-9f41-de435da8b942" (UID: "971e7680-ff0f-4010-9f41-de435da8b942"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:38:18.571050 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.570964    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971e7680-ff0f-4010-9f41-de435da8b942-kube-api-access-5cpdg" (OuterVolumeSpecName: "kube-api-access-5cpdg") pod "971e7680-ff0f-4010-9f41-de435da8b942" (UID: "971e7680-ff0f-4010-9f41-de435da8b942"). InnerVolumeSpecName "kube-api-access-5cpdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:38:18.578187 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.578154    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971e7680-ff0f-4010-9f41-de435da8b942-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "971e7680-ff0f-4010-9f41-de435da8b942" (UID: "971e7680-ff0f-4010-9f41-de435da8b942"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:38:18.669440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.669362    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971e7680-ff0f-4010-9f41-de435da8b942-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:38:18.669440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.669394    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpdg\" (UniqueName: \"kubernetes.io/projected/971e7680-ff0f-4010-9f41-de435da8b942-kube-api-access-5cpdg\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:38:18.669440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.669405    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971e7680-ff0f-4010-9f41-de435da8b942-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:38:18.669440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:18.669416    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971e7680-ff0f-4010-9f41-de435da8b942-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:38:19.228341 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.228310    2549 generic.go:358] "Generic (PLEG): container finished" podID="971e7680-ff0f-4010-9f41-de435da8b942" containerID="dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3" exitCode=0
May 25 13:38:19.228521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.228370    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" event={"ID":"971e7680-ff0f-4010-9f41-de435da8b942","Type":"ContainerDied","Data":"dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3"}
May 25 13:38:19.228521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.228397    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz" event={"ID":"971e7680-ff0f-4010-9f41-de435da8b942","Type":"ContainerDied","Data":"46cfcf9119cdd8682a2cdf63a691468f1c7b5ce97ec038dad52b35d5386f1121"}
May 25 13:38:19.228521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.228399    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"
May 25 13:38:19.228521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.228412    2549 scope.go:117] "RemoveContainer" containerID="4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48"
May 25 13:38:19.236155 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.235894    2549 scope.go:117] "RemoveContainer" containerID="dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3"
May 25 13:38:19.242726 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.242702    2549 scope.go:117] "RemoveContainer" containerID="92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae"
May 25 13:38:19.249059 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.249035    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"]
May 25 13:38:19.249873 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.249861    2549 scope.go:117] "RemoveContainer" containerID="4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48"
May 25 13:38:19.250129 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:38:19.250112    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48\": container with ID starting with 4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48 not found: ID does not exist" containerID="4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48"
May 25 13:38:19.250203 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.250135    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48"} err="failed to get container status \"4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48\": rpc error: code = NotFound desc = could not find container \"4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48\": container with ID starting with 4eb34a16b1e84b333b32cc190150888002a7fb146deae9b5597f3801d70f2b48 not found: ID does not exist"
May 25 13:38:19.250203 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.250152    2549 scope.go:117] "RemoveContainer" containerID="dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3"
May 25 13:38:19.250424 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:38:19.250403    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3\": container with ID starting with dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3 not found: ID does not exist" containerID="dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3"
May 25 13:38:19.250604 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.250428    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3"} err="failed to get container status \"dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3\": rpc error: code = NotFound desc = could not find container \"dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3\": container with ID starting with dd0d1fe700069e3e387ee35bd87c9efd0ca6d99c12e8a953c8d6812a5a76faf3 not found: ID does not exist"
May 25 13:38:19.250604 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.250444    2549 scope.go:117] "RemoveContainer" containerID="92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae"
May 25 13:38:19.250700 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:38:19.250662    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae\": container with ID starting with 92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae not found: ID does not exist" containerID="92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae"
May 25 13:38:19.250700 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.250680    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae"} err="failed to get container status \"92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae\": rpc error: code = NotFound desc = could not find container \"92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae\": container with ID starting with 92e15d14a73b1831c9886cc68e2ac17883f9ac3ec9744140821fe16bd032e4ae not found: ID does not exist"
May 25 13:38:19.254622 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:19.254597    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-gj2dz"]
May 25 13:38:20.233395 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:20.233357    2549 generic.go:358] "Generic (PLEG): container finished" podID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerID="cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d" exitCode=0
May 25 13:38:20.233395 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:20.233398    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" event={"ID":"47039c40-ae42-4a88-97a7-9666f15db2ff","Type":"ContainerDied","Data":"cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d"}
May 25 13:38:20.496069 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:20.495988    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971e7680-ff0f-4010-9f41-de435da8b942" path="/var/lib/kubelet/pods/971e7680-ff0f-4010-9f41-de435da8b942/volumes"
May 25 13:38:27.256178 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:27.256150    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" event={"ID":"47039c40-ae42-4a88-97a7-9666f15db2ff","Type":"ContainerStarted","Data":"cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48"}
May 25 13:38:28.260262 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:28.260226    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" event={"ID":"47039c40-ae42-4a88-97a7-9666f15db2ff","Type":"ContainerStarted","Data":"1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d"}
May 25 13:38:28.281517 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:28.281462    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podStartSLOduration=6.349751082 podStartE2EDuration="13.281446424s" podCreationTimestamp="2026-05-25 13:38:15 +0000 UTC" firstStartedPulling="2026-05-25 13:38:20.234510641 +0000 UTC m=+2050.273267742" lastFinishedPulling="2026-05-25 13:38:27.166205979 +0000 UTC m=+2057.204963084" observedRunningTime="2026-05-25 13:38:28.279677542 +0000 UTC m=+2058.318434669" watchObservedRunningTime="2026-05-25 13:38:28.281446424 +0000 UTC m=+2058.320203549"
May 25 13:38:29.263725 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:29.263692    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:29.263725 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:29.263726    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:29.265070 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:29.265041    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:38:30.266771 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:30.266734    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:38:35.271300 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:35.271257    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:38:35.271761 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:35.271733    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:38:45.272452 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:45.272409    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:38:55.272216 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:38:55.272170    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:39:05.272088 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:05.272046    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:39:11.121195 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:11.121168    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:39:11.124093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:11.124067    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:39:15.272104 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:15.272062    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:39:25.272167 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:25.272122    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:39:35.271774 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:35.271734    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 25 13:39:45.272485 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:45.272443    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:39:46.858594 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.858549    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"]
May 25 13:39:46.859064 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.858831    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container" containerID="cri-o://cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48" gracePeriod=30
May 25 13:39:46.859064 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.858871    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kube-rbac-proxy" containerID="cri-o://1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d" gracePeriod=30
May 25 13:39:46.966308 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966250    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"]
May 25 13:39:46.966567 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966555    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="storage-initializer"
May 25 13:39:46.966616 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966569    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="storage-initializer"
May 25 13:39:46.966616 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966582    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kube-rbac-proxy"
May 25 13:39:46.966616 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966588    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kube-rbac-proxy"
May 25 13:39:46.966616 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966596    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container"
May 25 13:39:46.966616 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966601    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container"
May 25 13:39:46.966774 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966650    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kube-rbac-proxy"
May 25 13:39:46.966774 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.966660    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="971e7680-ff0f-4010-9f41-de435da8b942" containerName="kserve-container"
May 25 13:39:46.969754 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.969731    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:46.971966 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.971944    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
May 25 13:39:46.972092 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.971944    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
May 25 13:39:46.978289 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:46.978236    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"]
May 25 13:39:47.132881 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.132849    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae126930-51fa-45aa-9be7-b981c0a49df7-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.133063 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.132901    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae126930-51fa-45aa-9be7-b981c0a49df7-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.133063 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.132933    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9878\" (UniqueName: \"kubernetes.io/projected/ae126930-51fa-45aa-9be7-b981c0a49df7-kube-api-access-j9878\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.133063 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.132996    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.234282 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.234215    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae126930-51fa-45aa-9be7-b981c0a49df7-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.234467 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.234295    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae126930-51fa-45aa-9be7-b981c0a49df7-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.234467 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.234330    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j9878\" (UniqueName: \"kubernetes.io/projected/ae126930-51fa-45aa-9be7-b981c0a49df7-kube-api-access-j9878\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.234467 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.234406    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.234679 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:39:47.234530    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
May 25 13:39:47.234679 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:39:47.234614    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls podName:ae126930-51fa-45aa-9be7-b981c0a49df7 nodeName:}" failed. No retries permitted until 2026-05-25 13:39:47.734589725 +0000 UTC m=+2137.773346828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls") pod "isvc-pmml-runtime-predictor-67bc544947-c5pjw" (UID: "ae126930-51fa-45aa-9be7-b981c0a49df7") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
May 25 13:39:47.234763 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.234748    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae126930-51fa-45aa-9be7-b981c0a49df7-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.234972 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.234950    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae126930-51fa-45aa-9be7-b981c0a49df7-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.246644 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.246614    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9878\" (UniqueName: \"kubernetes.io/projected/ae126930-51fa-45aa-9be7-b981c0a49df7-kube-api-access-j9878\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.476653 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.476559    2549 generic.go:358] "Generic (PLEG): container finished" podID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerID="1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d" exitCode=2
May 25 13:39:47.476653 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.476640    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" event={"ID":"47039c40-ae42-4a88-97a7-9666f15db2ff","Type":"ContainerDied","Data":"1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d"}
May 25 13:39:47.739308 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.739164    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.741764 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.741733    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-c5pjw\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:47.880294 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:47.880229    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:48.010196 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:48.010110    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"]
May 25 13:39:48.013607 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:39:48.013569    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae126930_51fa_45aa_9be7_b981c0a49df7.slice/crio-c80b561e2d8467622b970667bf8408801861e5de5120314589a54129550dafd3 WatchSource:0}: Error finding container c80b561e2d8467622b970667bf8408801861e5de5120314589a54129550dafd3: Status 404 returned error can't find the container with id c80b561e2d8467622b970667bf8408801861e5de5120314589a54129550dafd3
May 25 13:39:48.480758 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:48.480717    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" event={"ID":"ae126930-51fa-45aa-9be7-b981c0a49df7","Type":"ContainerStarted","Data":"d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d"}
May 25 13:39:48.480758 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:48.480758    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" event={"ID":"ae126930-51fa-45aa-9be7-b981c0a49df7","Type":"ContainerStarted","Data":"c80b561e2d8467622b970667bf8408801861e5de5120314589a54129550dafd3"}
May 25 13:39:50.267973 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.267921    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
May 25 13:39:50.709183 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.709157    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:39:50.865809 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.865778    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47039c40-ae42-4a88-97a7-9666f15db2ff-proxy-tls\") pod \"47039c40-ae42-4a88-97a7-9666f15db2ff\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") "
May 25 13:39:50.865998 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.865830    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvflj\" (UniqueName: \"kubernetes.io/projected/47039c40-ae42-4a88-97a7-9666f15db2ff-kube-api-access-xvflj\") pod \"47039c40-ae42-4a88-97a7-9666f15db2ff\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") "
May 25 13:39:50.865998 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.865863    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47039c40-ae42-4a88-97a7-9666f15db2ff-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"47039c40-ae42-4a88-97a7-9666f15db2ff\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") "
May 25 13:39:50.865998 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.865896    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47039c40-ae42-4a88-97a7-9666f15db2ff-kserve-provision-location\") pod \"47039c40-ae42-4a88-97a7-9666f15db2ff\" (UID: \"47039c40-ae42-4a88-97a7-9666f15db2ff\") "
May 25 13:39:50.866233 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.866211    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47039c40-ae42-4a88-97a7-9666f15db2ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47039c40-ae42-4a88-97a7-9666f15db2ff" (UID: "47039c40-ae42-4a88-97a7-9666f15db2ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:39:50.866318 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.866260    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47039c40-ae42-4a88-97a7-9666f15db2ff-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "47039c40-ae42-4a88-97a7-9666f15db2ff" (UID: "47039c40-ae42-4a88-97a7-9666f15db2ff"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:39:50.868175 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.868154    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47039c40-ae42-4a88-97a7-9666f15db2ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47039c40-ae42-4a88-97a7-9666f15db2ff" (UID: "47039c40-ae42-4a88-97a7-9666f15db2ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:39:50.868295 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.868211    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47039c40-ae42-4a88-97a7-9666f15db2ff-kube-api-access-xvflj" (OuterVolumeSpecName: "kube-api-access-xvflj") pod "47039c40-ae42-4a88-97a7-9666f15db2ff" (UID: "47039c40-ae42-4a88-97a7-9666f15db2ff"). InnerVolumeSpecName "kube-api-access-xvflj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:39:50.966716 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.966679    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47039c40-ae42-4a88-97a7-9666f15db2ff-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:39:50.966716 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.966713    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvflj\" (UniqueName: \"kubernetes.io/projected/47039c40-ae42-4a88-97a7-9666f15db2ff-kube-api-access-xvflj\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:39:50.966716 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.966726    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47039c40-ae42-4a88-97a7-9666f15db2ff-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:39:50.966959 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:50.966736    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47039c40-ae42-4a88-97a7-9666f15db2ff-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:39:51.490644 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.490610    2549 generic.go:358] "Generic (PLEG): container finished" podID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerID="cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48" exitCode=0
May 25 13:39:51.491134 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.490650    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" event={"ID":"47039c40-ae42-4a88-97a7-9666f15db2ff","Type":"ContainerDied","Data":"cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48"}
May 25 13:39:51.491134 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.490678    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6" event={"ID":"47039c40-ae42-4a88-97a7-9666f15db2ff","Type":"ContainerDied","Data":"f1d323947da20de810768d3d4e622462e8405d9872355abd3f56ec186c2cd8ab"}
May 25 13:39:51.491134 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.490699    2549 scope.go:117] "RemoveContainer" containerID="1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d"
May 25 13:39:51.491134 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.490714    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"
May 25 13:39:51.499129 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.499110    2549 scope.go:117] "RemoveContainer" containerID="cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48"
May 25 13:39:51.506644 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.506628    2549 scope.go:117] "RemoveContainer" containerID="cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d"
May 25 13:39:51.512358 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.512336    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"]
May 25 13:39:51.513821 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.513807    2549 scope.go:117] "RemoveContainer" containerID="1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d"
May 25 13:39:51.514067 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:39:51.514047    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d\": container with ID starting with 1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d not found: ID does not exist" containerID="1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d"
May 25 13:39:51.514127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.514074    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d"} err="failed to get container status \"1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d\": rpc error: code = NotFound desc = could not find container \"1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d\": container with ID starting with 1baecab97d36185db0ae0e084c0794b722586afbe2c3bbd0e779063a54a2e05d not found: ID does not exist"
May 25 13:39:51.514127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.514092    2549 scope.go:117] "RemoveContainer" containerID="cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48"
May 25 13:39:51.514326 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:39:51.514302    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48\": container with ID starting with cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48 not found: ID does not exist" containerID="cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48"
May 25 13:39:51.514415 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.514329    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48"} err="failed to get container status \"cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48\": rpc error: code = NotFound desc = could not find container \"cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48\": container with ID starting with cdbe9c2d1eb39a5aeef90463edbe88e87c619d229ff9c2198646fbb7f364fa48 not found: ID does not exist"
May 25 13:39:51.514415 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.514345    2549 scope.go:117] "RemoveContainer" containerID="cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d"
May 25 13:39:51.514535 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:39:51.514515    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d\": container with ID starting with cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d not found: ID does not exist" containerID="cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d"
May 25 13:39:51.514582 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.514541    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d"} err="failed to get container status \"cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d\": rpc error: code = NotFound desc = could not find container \"cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d\": container with ID starting with cc723c8230877154f5c934b892894fca36318f8e06b4713ff11cfdc13749373d not found: ID does not exist"
May 25 13:39:51.518396 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:51.518375    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-rtct6"]
May 25 13:39:52.494510 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:52.494476    2549 generic.go:358] "Generic (PLEG): container finished" podID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerID="d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d" exitCode=0
May 25 13:39:52.495441 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:52.495421    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" path="/var/lib/kubelet/pods/47039c40-ae42-4a88-97a7-9666f15db2ff/volumes"
May 25 13:39:52.495755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:52.495737    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" event={"ID":"ae126930-51fa-45aa-9be7-b981c0a49df7","Type":"ContainerDied","Data":"d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d"}
May 25 13:39:53.499955 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:53.499918    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" event={"ID":"ae126930-51fa-45aa-9be7-b981c0a49df7","Type":"ContainerStarted","Data":"4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194"}
May 25 13:39:53.499955 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:53.499960    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" event={"ID":"ae126930-51fa-45aa-9be7-b981c0a49df7","Type":"ContainerStarted","Data":"88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe"}
May 25 13:39:53.500399 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:53.500165    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:53.518435 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:53.518383    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podStartSLOduration=7.518368666 podStartE2EDuration="7.518368666s" podCreationTimestamp="2026-05-25 13:39:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:39:53.517171516 +0000 UTC m=+2143.555928640" watchObservedRunningTime="2026-05-25 13:39:53.518368666 +0000 UTC m=+2143.557125789"
May 25 13:39:54.502440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:54.502409    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:39:54.503727 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:54.503703    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:39:55.505293 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:39:55.505229    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:40:00.509292 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:40:00.509235    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:40:00.509924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:40:00.509856    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:40:10.510012 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:40:10.509968    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:40:20.509846 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:40:20.509802    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:40:30.510594 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:40:30.510555    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:40:40.510304 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:40:40.510243    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:40:50.510576 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:40:50.510533    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:41:00.509832 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:00.509788    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:41:10.510748 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:10.510709    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:41:18.496138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:18.496104    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:41:27.960762 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:27.960727    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"]
May 25 13:41:27.961362 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:27.961029    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" containerID="cri-o://88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe" gracePeriod=30
May 25 13:41:27.961362 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:27.961092    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kube-rbac-proxy" containerID="cri-o://4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194" gracePeriod=30
May 25 13:41:28.078185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078148    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"]
May 25 13:41:28.078579 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078560    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="storage-initializer"
May 25 13:41:28.078731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078581    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="storage-initializer"
May 25 13:41:28.078731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078684    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kube-rbac-proxy"
May 25 13:41:28.078731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078695    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kube-rbac-proxy"
May 25 13:41:28.078731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078712    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container"
May 25 13:41:28.078731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078721    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container"
May 25 13:41:28.078993 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078798    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kube-rbac-proxy"
May 25 13:41:28.078993 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.078813    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="47039c40-ae42-4a88-97a7-9666f15db2ff" containerName="kserve-container"
May 25 13:41:28.081777 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.081755    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.083709 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.083685    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
May 25 13:41:28.083884 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.083867    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 13:41:28.090215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.090192    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"]
May 25 13:41:28.185990 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.185952    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.186172 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.185998    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.186172 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.186030    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.186172 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.186062    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46ln7\" (UniqueName: \"kubernetes.io/projected/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kube-api-access-46ln7\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.287562 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.287471    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.287562 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.287520    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.287562 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.287548    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.287813 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.287571    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-46ln7\" (UniqueName: \"kubernetes.io/projected/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kube-api-access-46ln7\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.287884 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.287864    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.288159 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.288131    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.290247 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.290210    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.296692 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.296666    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46ln7\" (UniqueName: \"kubernetes.io/projected/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kube-api-access-46ln7\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.391693 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.391662    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:28.493211 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.493168    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 13:41:28.524603 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.524578    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"]
May 25 13:41:28.527078 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:41:28.527056    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9ae0fd_7f3b_488a_a9de_70afeeefb625.slice/crio-1a4a1f660b9486ab807a67ccebd1aeb5bb7eee95784e199e94701a414cf763a1 WatchSource:0}: Error finding container 1a4a1f660b9486ab807a67ccebd1aeb5bb7eee95784e199e94701a414cf763a1: Status 404 returned error can't find the container with id 1a4a1f660b9486ab807a67ccebd1aeb5bb7eee95784e199e94701a414cf763a1
May 25 13:41:28.528839 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.528821    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:41:28.762833 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.762787    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" event={"ID":"3a9ae0fd-7f3b-488a-a9de-70afeeefb625","Type":"ContainerStarted","Data":"e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2"}
May 25 13:41:28.762833 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.762838    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" event={"ID":"3a9ae0fd-7f3b-488a-a9de-70afeeefb625","Type":"ContainerStarted","Data":"1a4a1f660b9486ab807a67ccebd1aeb5bb7eee95784e199e94701a414cf763a1"}
May 25 13:41:28.764725 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.764697    2549 generic.go:358] "Generic (PLEG): container finished" podID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerID="4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194" exitCode=2
May 25 13:41:28.764844 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:28.764738    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" event={"ID":"ae126930-51fa-45aa-9be7-b981c0a49df7","Type":"ContainerDied","Data":"4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194"}
May 25 13:41:30.506048 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:30.506005    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
May 25 13:41:31.703411 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.703384    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:41:31.775649 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.775557    2549 generic.go:358] "Generic (PLEG): container finished" podID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerID="88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe" exitCode=0
May 25 13:41:31.775790 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.775648    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"
May 25 13:41:31.775790 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.775669    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" event={"ID":"ae126930-51fa-45aa-9be7-b981c0a49df7","Type":"ContainerDied","Data":"88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe"}
May 25 13:41:31.775790 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.775705    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw" event={"ID":"ae126930-51fa-45aa-9be7-b981c0a49df7","Type":"ContainerDied","Data":"c80b561e2d8467622b970667bf8408801861e5de5120314589a54129550dafd3"}
May 25 13:41:31.775790 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.775726    2549 scope.go:117] "RemoveContainer" containerID="4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194"
May 25 13:41:31.783189 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.783170    2549 scope.go:117] "RemoveContainer" containerID="88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe"
May 25 13:41:31.790343 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.790321    2549 scope.go:117] "RemoveContainer" containerID="d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d"
May 25 13:41:31.796971 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.796952    2549 scope.go:117] "RemoveContainer" containerID="4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194"
May 25 13:41:31.797240 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:41:31.797220    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194\": container with ID starting with 4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194 not found: ID does not exist" containerID="4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194"
May 25 13:41:31.797321 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.797248    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194"} err="failed to get container status \"4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194\": rpc error: code = NotFound desc = could not find container \"4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194\": container with ID starting with 4c724bbac1ea6005c96dc725c2a6b398023e553bdfaa82ccd7c2e6320053e194 not found: ID does not exist"
May 25 13:41:31.797321 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.797293    2549 scope.go:117] "RemoveContainer" containerID="88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe"
May 25 13:41:31.797540 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:41:31.797523    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe\": container with ID starting with 88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe not found: ID does not exist" containerID="88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe"
May 25 13:41:31.797583 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.797558    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe"} err="failed to get container status \"88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe\": rpc error: code = NotFound desc = could not find container \"88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe\": container with ID starting with 88aa51c2474f1d23f091a78e55f8876a74ec7baa6627e9d1c2e8fe3fe63378fe not found: ID does not exist"
May 25 13:41:31.797583 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.797575    2549 scope.go:117] "RemoveContainer" containerID="d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d"
May 25 13:41:31.797790 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:41:31.797773    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d\": container with ID starting with d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d not found: ID does not exist" containerID="d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d"
May 25 13:41:31.797830 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.797794    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d"} err="failed to get container status \"d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d\": rpc error: code = NotFound desc = could not find container \"d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d\": container with ID starting with d71a3120c36f0307964877a577ac26a4e975ad03fe02f59f6fb8d8e806454e8d not found: ID does not exist"
May 25 13:41:31.814075 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.814051    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9878\" (UniqueName: \"kubernetes.io/projected/ae126930-51fa-45aa-9be7-b981c0a49df7-kube-api-access-j9878\") pod \"ae126930-51fa-45aa-9be7-b981c0a49df7\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") "
May 25 13:41:31.814167 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.814093    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae126930-51fa-45aa-9be7-b981c0a49df7-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"ae126930-51fa-45aa-9be7-b981c0a49df7\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") "
May 25 13:41:31.814255 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.814237    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls\") pod \"ae126930-51fa-45aa-9be7-b981c0a49df7\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") "
May 25 13:41:31.814350 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.814336    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae126930-51fa-45aa-9be7-b981c0a49df7-kserve-provision-location\") pod \"ae126930-51fa-45aa-9be7-b981c0a49df7\" (UID: \"ae126930-51fa-45aa-9be7-b981c0a49df7\") "
May 25 13:41:31.814428 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.814403    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae126930-51fa-45aa-9be7-b981c0a49df7-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "ae126930-51fa-45aa-9be7-b981c0a49df7" (UID: "ae126930-51fa-45aa-9be7-b981c0a49df7"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:41:31.814623 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.814602    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae126930-51fa-45aa-9be7-b981c0a49df7-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:41:31.814696 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.814599    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae126930-51fa-45aa-9be7-b981c0a49df7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae126930-51fa-45aa-9be7-b981c0a49df7" (UID: "ae126930-51fa-45aa-9be7-b981c0a49df7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:41:31.816226 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.816207    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae126930-51fa-45aa-9be7-b981c0a49df7" (UID: "ae126930-51fa-45aa-9be7-b981c0a49df7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:41:31.816305 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.816254    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae126930-51fa-45aa-9be7-b981c0a49df7-kube-api-access-j9878" (OuterVolumeSpecName: "kube-api-access-j9878") pod "ae126930-51fa-45aa-9be7-b981c0a49df7" (UID: "ae126930-51fa-45aa-9be7-b981c0a49df7"). InnerVolumeSpecName "kube-api-access-j9878". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:41:31.915882 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.915834    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae126930-51fa-45aa-9be7-b981c0a49df7-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:41:31.915882 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.915874    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae126930-51fa-45aa-9be7-b981c0a49df7-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:41:31.915882 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:31.915884    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9878\" (UniqueName: \"kubernetes.io/projected/ae126930-51fa-45aa-9be7-b981c0a49df7-kube-api-access-j9878\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:41:32.096952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:32.096921    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"]
May 25 13:41:32.100830 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:32.100801    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-c5pjw"]
May 25 13:41:32.495583 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:32.495547    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" path="/var/lib/kubelet/pods/ae126930-51fa-45aa-9be7-b981c0a49df7/volumes"
May 25 13:41:32.779110 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:32.779073    2549 generic.go:358] "Generic (PLEG): container finished" podID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerID="e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2" exitCode=0
May 25 13:41:32.779596 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:32.779148    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" event={"ID":"3a9ae0fd-7f3b-488a-a9de-70afeeefb625","Type":"ContainerDied","Data":"e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2"}
May 25 13:41:33.789686 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:33.789646    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" event={"ID":"3a9ae0fd-7f3b-488a-a9de-70afeeefb625","Type":"ContainerStarted","Data":"e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843"}
May 25 13:41:33.790080 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:33.789694    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" event={"ID":"3a9ae0fd-7f3b-488a-a9de-70afeeefb625","Type":"ContainerStarted","Data":"3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300"}
May 25 13:41:33.790080 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:33.790024    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:33.809284 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:33.809212    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podStartSLOduration=5.809198432 podStartE2EDuration="5.809198432s" podCreationTimestamp="2026-05-25 13:41:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:41:33.807717114 +0000 UTC m=+2243.846474238" watchObservedRunningTime="2026-05-25 13:41:33.809198432 +0000 UTC m=+2243.847955555"
May 25 13:41:34.792973 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:34.792934    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:34.794115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:34.794083    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:41:35.795827 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:35.795788    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:41:40.800459 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:40.800426    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:41:40.801074 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:40.801046    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:41:50.801471 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:41:50.801428    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:42:00.801957 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:00.801916    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:42:10.801761 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:10.801723    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:42:20.801732 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:20.801689    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:42:30.801867 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:30.801820    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:42:40.801159 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:40.801113    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:42:42.491925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:42.491877    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:42:52.495434 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:52.495353    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:42:59.066521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.066481    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"]
May 25 13:42:59.066912 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.066803    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" containerID="cri-o://3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300" gracePeriod=30
May 25 13:42:59.066981 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.066894    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kube-rbac-proxy" containerID="cri-o://e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843" gracePeriod=30
May 25 13:42:59.191715 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.191664    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"]
May 25 13:42:59.191965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.191933    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="storage-initializer"
May 25 13:42:59.191965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.191946    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="storage-initializer"
May 25 13:42:59.191965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.191959    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kube-rbac-proxy"
May 25 13:42:59.191965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.191969    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kube-rbac-proxy"
May 25 13:42:59.192166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.191982    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container"
May 25 13:42:59.192166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.191988    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container"
May 25 13:42:59.192166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.192031    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kserve-container"
May 25 13:42:59.192166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.192041    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ae126930-51fa-45aa-9be7-b981c0a49df7" containerName="kube-rbac-proxy"
May 25 13:42:59.194861 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.194842    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.197141 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.197118    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d79fbd-predictor-serving-cert\""
May 25 13:42:59.197404 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.197388    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d79fbd-kube-rbac-proxy-sar-config\""
May 25 13:42:59.205285 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.205245    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"]
May 25 13:42:59.285847 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.285811    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82597652-966b-4713-a591-e3d20a02c6f8-proxy-tls\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.286031 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.285858    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82597652-966b-4713-a591-e3d20a02c6f8-kserve-provision-location\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.286031 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.285971    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82597652-966b-4713-a591-e3d20a02c6f8-isvc-primary-d79fbd-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.286031 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.286012    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrt8\" (UniqueName: \"kubernetes.io/projected/82597652-966b-4713-a591-e3d20a02c6f8-kube-api-access-ftrt8\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.387329 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.387298    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82597652-966b-4713-a591-e3d20a02c6f8-proxy-tls\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.387527 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.387347    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82597652-966b-4713-a591-e3d20a02c6f8-kserve-provision-location\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.387527 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.387398    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82597652-966b-4713-a591-e3d20a02c6f8-isvc-primary-d79fbd-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.387527 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.387419    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrt8\" (UniqueName: \"kubernetes.io/projected/82597652-966b-4713-a591-e3d20a02c6f8-kube-api-access-ftrt8\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.387828 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.387805    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82597652-966b-4713-a591-e3d20a02c6f8-kserve-provision-location\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.388091 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.388068    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82597652-966b-4713-a591-e3d20a02c6f8-isvc-primary-d79fbd-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.390023 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.390000    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82597652-966b-4713-a591-e3d20a02c6f8-proxy-tls\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.396464 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.396440    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrt8\" (UniqueName: \"kubernetes.io/projected/82597652-966b-4713-a591-e3d20a02c6f8-kube-api-access-ftrt8\") pod \"isvc-primary-d79fbd-predictor-64c4c46948-xhdsc\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") " pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.508003 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.507963    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:42:59.639144 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:42:59.639062    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"]
May 25 13:42:59.642286 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:42:59.642248    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82597652_966b_4713_a591_e3d20a02c6f8.slice/crio-7075009aa2f1599f87f9993cde2fd01b7971afed47f8d7507e399f695e0829ee WatchSource:0}: Error finding container 7075009aa2f1599f87f9993cde2fd01b7971afed47f8d7507e399f695e0829ee: Status 404 returned error can't find the container with id 7075009aa2f1599f87f9993cde2fd01b7971afed47f8d7507e399f695e0829ee
May 25 13:43:00.038163 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:00.038061    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" event={"ID":"82597652-966b-4713-a591-e3d20a02c6f8","Type":"ContainerStarted","Data":"965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9"}
May 25 13:43:00.038163 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:00.038107    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" event={"ID":"82597652-966b-4713-a591-e3d20a02c6f8","Type":"ContainerStarted","Data":"7075009aa2f1599f87f9993cde2fd01b7971afed47f8d7507e399f695e0829ee"}
May 25 13:43:00.040046 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:00.040016    2549 generic.go:358] "Generic (PLEG): container finished" podID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerID="e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843" exitCode=2
May 25 13:43:00.040153 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:00.040079    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" event={"ID":"3a9ae0fd-7f3b-488a-a9de-70afeeefb625","Type":"ContainerDied","Data":"e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843"}
May 25 13:43:00.796393 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:00.796341    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
May 25 13:43:02.492472 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.492428    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 13:43:02.613702 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.613678    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:43:02.713161 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.713063    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46ln7\" (UniqueName: \"kubernetes.io/projected/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kube-api-access-46ln7\") pod \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") "
May 25 13:43:02.713161 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.713143    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-proxy-tls\") pod \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") "
May 25 13:43:02.713441 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.713255    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") "
May 25 13:43:02.713441 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.713359    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kserve-provision-location\") pod \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\" (UID: \"3a9ae0fd-7f3b-488a-a9de-70afeeefb625\") "
May 25 13:43:02.713646 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.713621    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "3a9ae0fd-7f3b-488a-a9de-70afeeefb625" (UID: "3a9ae0fd-7f3b-488a-a9de-70afeeefb625"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:43:02.713703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.713654    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a9ae0fd-7f3b-488a-a9de-70afeeefb625" (UID: "3a9ae0fd-7f3b-488a-a9de-70afeeefb625"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:43:02.715345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.715317    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a9ae0fd-7f3b-488a-a9de-70afeeefb625" (UID: "3a9ae0fd-7f3b-488a-a9de-70afeeefb625"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:43:02.715345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.715334    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kube-api-access-46ln7" (OuterVolumeSpecName: "kube-api-access-46ln7") pod "3a9ae0fd-7f3b-488a-a9de-70afeeefb625" (UID: "3a9ae0fd-7f3b-488a-a9de-70afeeefb625"). InnerVolumeSpecName "kube-api-access-46ln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:43:02.814862 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.814804    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:43:02.814862 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.814855    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:43:02.814862 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.814867    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:43:02.814862 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:02.814878    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46ln7\" (UniqueName: \"kubernetes.io/projected/3a9ae0fd-7f3b-488a-a9de-70afeeefb625-kube-api-access-46ln7\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:43:03.048837 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.048737    2549 generic.go:358] "Generic (PLEG): container finished" podID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerID="3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300" exitCode=0
May 25 13:43:03.048837 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.048808    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" event={"ID":"3a9ae0fd-7f3b-488a-a9de-70afeeefb625","Type":"ContainerDied","Data":"3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300"}
May 25 13:43:03.048837 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.048814    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"
May 25 13:43:03.049093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.048843    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2" event={"ID":"3a9ae0fd-7f3b-488a-a9de-70afeeefb625","Type":"ContainerDied","Data":"1a4a1f660b9486ab807a67ccebd1aeb5bb7eee95784e199e94701a414cf763a1"}
May 25 13:43:03.049093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.048865    2549 scope.go:117] "RemoveContainer" containerID="e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843"
May 25 13:43:03.057338 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.057311    2549 scope.go:117] "RemoveContainer" containerID="3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300"
May 25 13:43:03.069026 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.069003    2549 scope.go:117] "RemoveContainer" containerID="e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2"
May 25 13:43:03.071159 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.071136    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"]
May 25 13:43:03.076095 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.076076    2549 scope.go:117] "RemoveContainer" containerID="e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843"
May 25 13:43:03.076397 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:43:03.076376    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843\": container with ID starting with e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843 not found: ID does not exist" containerID="e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843"
May 25 13:43:03.076470 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.076405    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843"} err="failed to get container status \"e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843\": rpc error: code = NotFound desc = could not find container \"e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843\": container with ID starting with e021abc7f7825358816b79f03d30f380114b6ae5ca99178e6cd081bae4047843 not found: ID does not exist"
May 25 13:43:03.076470 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.076424    2549 scope.go:117] "RemoveContainer" containerID="3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300"
May 25 13:43:03.076676 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:43:03.076659    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300\": container with ID starting with 3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300 not found: ID does not exist" containerID="3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300"
May 25 13:43:03.076726 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.076683    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300"} err="failed to get container status \"3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300\": rpc error: code = NotFound desc = could not find container \"3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300\": container with ID starting with 3ba8bb47f1d6713c4fa8bf7def39b5e5a36d6a7c8c7ef96c82de02c07d381300 not found: ID does not exist"
May 25 13:43:03.076726 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.076700    2549 scope.go:117] "RemoveContainer" containerID="e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2"
May 25 13:43:03.076910 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:43:03.076891    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2\": container with ID starting with e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2 not found: ID does not exist" containerID="e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2"
May 25 13:43:03.076951 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.076916    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2"} err="failed to get container status \"e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2\": rpc error: code = NotFound desc = could not find container \"e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2\": container with ID starting with e8aba89e417475e3e1c788a6cb16a73906691199a972712d581ead91e82be6f2 not found: ID does not exist"
May 25 13:43:03.082750 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:03.082727    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-rrwp2"]
May 25 13:43:04.053380 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:04.053341    2549 generic.go:358] "Generic (PLEG): container finished" podID="82597652-966b-4713-a591-e3d20a02c6f8" containerID="965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9" exitCode=0
May 25 13:43:04.053816 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:04.053423    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" event={"ID":"82597652-966b-4713-a591-e3d20a02c6f8","Type":"ContainerDied","Data":"965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9"}
May 25 13:43:04.497063 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:04.497025    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" path="/var/lib/kubelet/pods/3a9ae0fd-7f3b-488a-a9de-70afeeefb625/volumes"
May 25 13:43:05.058040 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:05.058005    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" event={"ID":"82597652-966b-4713-a591-e3d20a02c6f8","Type":"ContainerStarted","Data":"3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2"}
May 25 13:43:05.058558 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:05.058050    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" event={"ID":"82597652-966b-4713-a591-e3d20a02c6f8","Type":"ContainerStarted","Data":"e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c"}
May 25 13:43:05.058558 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:05.058294    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:43:05.078874 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:05.078818    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podStartSLOduration=6.078799159 podStartE2EDuration="6.078799159s" podCreationTimestamp="2026-05-25 13:42:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:43:05.078127791 +0000 UTC m=+2335.116884916" watchObservedRunningTime="2026-05-25 13:43:05.078799159 +0000 UTC m=+2335.117556288"
May 25 13:43:06.061693 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:06.061660    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:43:06.062838 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:06.062810    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:43:07.064120 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:07.064074    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:43:12.068983 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:12.068951    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:43:12.069578 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:12.069550    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:43:22.070332 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:22.070293    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:43:32.070116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:32.070076    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:43:42.069554 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:42.069511    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:43:52.069879 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:43:52.069836    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:44:02.069827 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:02.069782    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 13:44:11.140326 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:11.140264    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:44:11.143045 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:11.143027    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:44:12.070398 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:12.070369    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:44:19.315334 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315240    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"]
May 25 13:44:19.315755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315507    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container"
May 25 13:44:19.315755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315519    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container"
May 25 13:44:19.315755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315528    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kube-rbac-proxy"
May 25 13:44:19.315755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315533    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kube-rbac-proxy"
May 25 13:44:19.315755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315550    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="storage-initializer"
May 25 13:44:19.315755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315555    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="storage-initializer"
May 25 13:44:19.315755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315596    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kube-rbac-proxy"
May 25 13:44:19.315755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.315605    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="3a9ae0fd-7f3b-488a-a9de-70afeeefb625" containerName="kserve-container"
May 25 13:44:19.318531 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.318502    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.320906 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.320882    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d79fbd-predictor-serving-cert\""
May 25 13:44:19.321042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.320910    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d79fbd-dockercfg-bgkdk\""
May 25 13:44:19.321042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.320882    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 13:44:19.321042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.320885    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d79fbd\""
May 25 13:44:19.321604 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.321589    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\""
May 25 13:44:19.328217 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.328179    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"]
May 25 13:44:19.496149 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.496104    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kserve-provision-location\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.496339 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.496156    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.496339 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.496229    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-cabundle-cert\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.496339 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.496261    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmc7d\" (UniqueName: \"kubernetes.io/projected/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kube-api-access-zmc7d\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.496339 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.496302    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.597340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.597224    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.597340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.597328    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-cabundle-cert\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.597605 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.597363    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zmc7d\" (UniqueName: \"kubernetes.io/projected/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kube-api-access-zmc7d\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.597605 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.597410    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.597605 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.597449    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kserve-provision-location\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.597605 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:19.597575    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-serving-cert: secret "isvc-secondary-d79fbd-predictor-serving-cert" not found
May 25 13:44:19.597803 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:19.597652    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls podName:35b62580-52fa-4038-92a8-9d43dcb1b5f8 nodeName:}" failed. No retries permitted until 2026-05-25 13:44:20.097632084 +0000 UTC m=+2410.136389203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls") pod "isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2" (UID: "35b62580-52fa-4038-92a8-9d43dcb1b5f8") : secret "isvc-secondary-d79fbd-predictor-serving-cert" not found
May 25 13:44:19.597803 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.597788    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kserve-provision-location\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.597978 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.597958    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.598016 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.597988    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-cabundle-cert\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:19.605876 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:19.605849    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmc7d\" (UniqueName: \"kubernetes.io/projected/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kube-api-access-zmc7d\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:20.102236 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:20.102186    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:20.104731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:20.104698    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls\") pod \"isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") " pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:20.228336 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:20.228263    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:20.358115 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:20.358027    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"]
May 25 13:44:20.361873 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:44:20.361843    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b62580_52fa_4038_92a8_9d43dcb1b5f8.slice/crio-236971338e1776ebe25dde198ab94933853aea506fafe8751da4d06b0ce25130 WatchSource:0}: Error finding container 236971338e1776ebe25dde198ab94933853aea506fafe8751da4d06b0ce25130: Status 404 returned error can't find the container with id 236971338e1776ebe25dde198ab94933853aea506fafe8751da4d06b0ce25130
May 25 13:44:21.273146 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:21.273113    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2" event={"ID":"35b62580-52fa-4038-92a8-9d43dcb1b5f8","Type":"ContainerStarted","Data":"b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a"}
May 25 13:44:21.273146 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:21.273150    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2" event={"ID":"35b62580-52fa-4038-92a8-9d43dcb1b5f8","Type":"ContainerStarted","Data":"236971338e1776ebe25dde198ab94933853aea506fafe8751da4d06b0ce25130"}
May 25 13:44:24.282447 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:24.282412    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_35b62580-52fa-4038-92a8-9d43dcb1b5f8/storage-initializer/0.log"
May 25 13:44:24.282841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:24.282457    2549 generic.go:358] "Generic (PLEG): container finished" podID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" containerID="b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a" exitCode=1
May 25 13:44:24.282841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:24.282542    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2" event={"ID":"35b62580-52fa-4038-92a8-9d43dcb1b5f8","Type":"ContainerDied","Data":"b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a"}
May 25 13:44:25.286408 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:25.286376    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_35b62580-52fa-4038-92a8-9d43dcb1b5f8/storage-initializer/0.log"
May 25 13:44:25.286811 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:25.286473    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2" event={"ID":"35b62580-52fa-4038-92a8-9d43dcb1b5f8","Type":"ContainerStarted","Data":"a874b40475d292fa79a996a2afc5f5e4bd36bc650adc5cfb33244ca97df6ec0c"}
May 25 13:44:30.301337 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:30.301304    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_35b62580-52fa-4038-92a8-9d43dcb1b5f8/storage-initializer/1.log"
May 25 13:44:30.301746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:30.301666    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_35b62580-52fa-4038-92a8-9d43dcb1b5f8/storage-initializer/0.log"
May 25 13:44:30.301746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:30.301696    2549 generic.go:358] "Generic (PLEG): container finished" podID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" containerID="a874b40475d292fa79a996a2afc5f5e4bd36bc650adc5cfb33244ca97df6ec0c" exitCode=1
May 25 13:44:30.301841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:30.301779    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2" event={"ID":"35b62580-52fa-4038-92a8-9d43dcb1b5f8","Type":"ContainerDied","Data":"a874b40475d292fa79a996a2afc5f5e4bd36bc650adc5cfb33244ca97df6ec0c"}
May 25 13:44:30.301841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:30.301825    2549 scope.go:117] "RemoveContainer" containerID="b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a"
May 25 13:44:30.302190 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:30.302173    2549 scope.go:117] "RemoveContainer" containerID="b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a"
May 25 13:44:30.312380 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:30.312348    2549 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_kserve-ci-e2e-test_35b62580-52fa-4038-92a8-9d43dcb1b5f8_0 in pod sandbox 236971338e1776ebe25dde198ab94933853aea506fafe8751da4d06b0ce25130 from index: no such id: 'b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a'" containerID="b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a"
May 25 13:44:30.312456 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:30.312395    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_kserve-ci-e2e-test_35b62580-52fa-4038-92a8-9d43dcb1b5f8_0 in pod sandbox 236971338e1776ebe25dde198ab94933853aea506fafe8751da4d06b0ce25130 from index: no such id: 'b863c4b1b0daec610405dd49c29472cb9ba1b290c716cd48805919833f3e406a'"
May 25 13:44:30.312624 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:30.312603    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_kserve-ci-e2e-test(35b62580-52fa-4038-92a8-9d43dcb1b5f8)\"" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2" podUID="35b62580-52fa-4038-92a8-9d43dcb1b5f8"
May 25 13:44:31.305943 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:31.305916    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_35b62580-52fa-4038-92a8-9d43dcb1b5f8/storage-initializer/1.log"
May 25 13:44:35.359080 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.359035    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"]
May 25 13:44:35.407218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.407182    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"]
May 25 13:44:35.407693 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.407517    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container" containerID="cri-o://e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c" gracePeriod=30
May 25 13:44:35.407693 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.407566    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kube-rbac-proxy" containerID="cri-o://3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2" gracePeriod=30
May 25 13:44:35.479198 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.479161    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"]
May 25 13:44:35.482972 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.482948    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.485502 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.485477    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fa253b-predictor-serving-cert\""
May 25 13:44:35.485627 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.485477    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-fa253b-dockercfg-6r744\""
May 25 13:44:35.485627 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.485482    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-fa253b\""
May 25 13:44:35.485726 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.485630    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\""
May 25 13:44:35.492708 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.492679    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"]
May 25 13:44:35.517660 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.517634    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kserve-provision-location\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.517789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.517685    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.517789 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.517753    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lswfr\" (UniqueName: \"kubernetes.io/projected/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kube-api-access-lswfr\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.517880 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.517856    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-cabundle-cert\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.517948 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.517929    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.526039 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.526021    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_35b62580-52fa-4038-92a8-9d43dcb1b5f8/storage-initializer/1.log"
May 25 13:44:35.526137 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.526085    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:35.619234 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619202    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmc7d\" (UniqueName: \"kubernetes.io/projected/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kube-api-access-zmc7d\") pod \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") "
May 25 13:44:35.619442 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619255    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\") pod \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") "
May 25 13:44:35.619442 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619311    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls\") pod \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") "
May 25 13:44:35.619442 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619352    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kserve-provision-location\") pod \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") "
May 25 13:44:35.619442 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619370    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-cabundle-cert\") pod \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\" (UID: \"35b62580-52fa-4038-92a8-9d43dcb1b5f8\") "
May 25 13:44:35.619651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619448    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kserve-provision-location\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.619651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619503    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.619651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619538    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lswfr\" (UniqueName: \"kubernetes.io/projected/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kube-api-access-lswfr\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.619651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619572    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-cabundle-cert\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.619918 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619626    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35b62580-52fa-4038-92a8-9d43dcb1b5f8" (UID: "35b62580-52fa-4038-92a8-9d43dcb1b5f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:35.619918 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619707    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.619918 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619775    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:35.619918 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619801    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35b62580-52fa-4038-92a8-9d43dcb1b5f8" (UID: "35b62580-52fa-4038-92a8-9d43dcb1b5f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:44:35.619918 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:35.619866    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-serving-cert: secret "isvc-init-fail-fa253b-predictor-serving-cert" not found
May 25 13:44:35.619918 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.619912    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kserve-provision-location\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.620226 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:35.619935    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls podName:b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49 nodeName:}" failed. No retries permitted until 2026-05-25 13:44:36.1199147 +0000 UTC m=+2426.158671802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls") pod "isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7" (UID: "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49") : secret "isvc-init-fail-fa253b-predictor-serving-cert" not found
May 25 13:44:35.620359 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.620335    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.620427 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.620343    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-cabundle-cert\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.620490 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.620465    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-isvc-secondary-d79fbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-d79fbd-kube-rbac-proxy-sar-config") pod "35b62580-52fa-4038-92a8-9d43dcb1b5f8" (UID: "35b62580-52fa-4038-92a8-9d43dcb1b5f8"). InnerVolumeSpecName "isvc-secondary-d79fbd-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:44:35.621670 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.621650    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kube-api-access-zmc7d" (OuterVolumeSpecName: "kube-api-access-zmc7d") pod "35b62580-52fa-4038-92a8-9d43dcb1b5f8" (UID: "35b62580-52fa-4038-92a8-9d43dcb1b5f8"). InnerVolumeSpecName "kube-api-access-zmc7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:44:35.621748 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.621703    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35b62580-52fa-4038-92a8-9d43dcb1b5f8" (UID: "35b62580-52fa-4038-92a8-9d43dcb1b5f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:44:35.628408 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.628377    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lswfr\" (UniqueName: \"kubernetes.io/projected/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kube-api-access-lswfr\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:35.721081 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.721044    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-isvc-secondary-d79fbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:35.721081 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.721078    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b62580-52fa-4038-92a8-9d43dcb1b5f8-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:35.721081 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.721088    2549 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b62580-52fa-4038-92a8-9d43dcb1b5f8-cabundle-cert\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:35.721081 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:35.721096    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmc7d\" (UniqueName: \"kubernetes.io/projected/35b62580-52fa-4038-92a8-9d43dcb1b5f8-kube-api-access-zmc7d\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:36.124903 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.124877    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:36.127452 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.127422    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls\") pod \"isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") " pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:36.321746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.321714    2549 generic.go:358] "Generic (PLEG): container finished" podID="82597652-966b-4713-a591-e3d20a02c6f8" containerID="3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2" exitCode=2
May 25 13:44:36.321924 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.321786    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" event={"ID":"82597652-966b-4713-a591-e3d20a02c6f8","Type":"ContainerDied","Data":"3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2"}
May 25 13:44:36.322812 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.322795    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2_35b62580-52fa-4038-92a8-9d43dcb1b5f8/storage-initializer/1.log"
May 25 13:44:36.322906 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.322848    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2" event={"ID":"35b62580-52fa-4038-92a8-9d43dcb1b5f8","Type":"ContainerDied","Data":"236971338e1776ebe25dde198ab94933853aea506fafe8751da4d06b0ce25130"}
May 25 13:44:36.322906 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.322873    2549 scope.go:117] "RemoveContainer" containerID="a874b40475d292fa79a996a2afc5f5e4bd36bc650adc5cfb33244ca97df6ec0c"
May 25 13:44:36.322906 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.322897    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"
May 25 13:44:36.362835 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.362792    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"]
May 25 13:44:36.368180 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.368151    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d79fbd-predictor-64c58d65f5-m8bw2"]
May 25 13:44:36.399036 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.398948    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:36.498111 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.498075    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" path="/var/lib/kubelet/pods/35b62580-52fa-4038-92a8-9d43dcb1b5f8/volumes"
May 25 13:44:36.525615 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:36.525587    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"]
May 25 13:44:36.527980 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:44:36.527950    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dcc9f8_8cc8_4cac_a0b0_b879fc9bad49.slice/crio-d6c2e358cd20c816134b4ac0e7f472d3c637299ba3fb27a0d9244e1c35d9a41a WatchSource:0}: Error finding container d6c2e358cd20c816134b4ac0e7f472d3c637299ba3fb27a0d9244e1c35d9a41a: Status 404 returned error can't find the container with id d6c2e358cd20c816134b4ac0e7f472d3c637299ba3fb27a0d9244e1c35d9a41a
May 25 13:44:37.064802 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:37.064764    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
May 25 13:44:37.327658 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:37.327564    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7" event={"ID":"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49","Type":"ContainerStarted","Data":"db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5"}
May 25 13:44:37.327658 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:37.327602    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7" event={"ID":"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49","Type":"ContainerStarted","Data":"d6c2e358cd20c816134b4ac0e7f472d3c637299ba3fb27a0d9244e1c35d9a41a"}
May 25 13:44:39.749805 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.749781    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:44:39.856087 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.855981    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82597652-966b-4713-a591-e3d20a02c6f8-isvc-primary-d79fbd-kube-rbac-proxy-sar-config\") pod \"82597652-966b-4713-a591-e3d20a02c6f8\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") "
May 25 13:44:39.856087 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.856034    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82597652-966b-4713-a591-e3d20a02c6f8-kserve-provision-location\") pod \"82597652-966b-4713-a591-e3d20a02c6f8\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") "
May 25 13:44:39.856087 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.856065    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftrt8\" (UniqueName: \"kubernetes.io/projected/82597652-966b-4713-a591-e3d20a02c6f8-kube-api-access-ftrt8\") pod \"82597652-966b-4713-a591-e3d20a02c6f8\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") "
May 25 13:44:39.856380 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.856244    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82597652-966b-4713-a591-e3d20a02c6f8-proxy-tls\") pod \"82597652-966b-4713-a591-e3d20a02c6f8\" (UID: \"82597652-966b-4713-a591-e3d20a02c6f8\") "
May 25 13:44:39.856442 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.856408    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82597652-966b-4713-a591-e3d20a02c6f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82597652-966b-4713-a591-e3d20a02c6f8" (UID: "82597652-966b-4713-a591-e3d20a02c6f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:39.856495 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.856458    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82597652-966b-4713-a591-e3d20a02c6f8-isvc-primary-d79fbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-d79fbd-kube-rbac-proxy-sar-config") pod "82597652-966b-4713-a591-e3d20a02c6f8" (UID: "82597652-966b-4713-a591-e3d20a02c6f8"). InnerVolumeSpecName "isvc-primary-d79fbd-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:44:39.856495 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.856488    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82597652-966b-4713-a591-e3d20a02c6f8-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:39.858351 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.858313    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82597652-966b-4713-a591-e3d20a02c6f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82597652-966b-4713-a591-e3d20a02c6f8" (UID: "82597652-966b-4713-a591-e3d20a02c6f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:44:39.858351 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.858320    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82597652-966b-4713-a591-e3d20a02c6f8-kube-api-access-ftrt8" (OuterVolumeSpecName: "kube-api-access-ftrt8") pod "82597652-966b-4713-a591-e3d20a02c6f8" (UID: "82597652-966b-4713-a591-e3d20a02c6f8"). InnerVolumeSpecName "kube-api-access-ftrt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:44:39.957409 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.957348    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-d79fbd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82597652-966b-4713-a591-e3d20a02c6f8-isvc-primary-d79fbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:39.957409 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.957406    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftrt8\" (UniqueName: \"kubernetes.io/projected/82597652-966b-4713-a591-e3d20a02c6f8-kube-api-access-ftrt8\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:39.957612 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:39.957417    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82597652-966b-4713-a591-e3d20a02c6f8-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:40.336821 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.336787    2549 generic.go:358] "Generic (PLEG): container finished" podID="82597652-966b-4713-a591-e3d20a02c6f8" containerID="e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c" exitCode=0
May 25 13:44:40.336976 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.336863    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"
May 25 13:44:40.336976 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.336878    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" event={"ID":"82597652-966b-4713-a591-e3d20a02c6f8","Type":"ContainerDied","Data":"e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c"}
May 25 13:44:40.336976 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.336913    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc" event={"ID":"82597652-966b-4713-a591-e3d20a02c6f8","Type":"ContainerDied","Data":"7075009aa2f1599f87f9993cde2fd01b7971afed47f8d7507e399f695e0829ee"}
May 25 13:44:40.336976 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.336929    2549 scope.go:117] "RemoveContainer" containerID="3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2"
May 25 13:44:40.345753 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.345727    2549 scope.go:117] "RemoveContainer" containerID="e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c"
May 25 13:44:40.352913 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.352894    2549 scope.go:117] "RemoveContainer" containerID="965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9"
May 25 13:44:40.359941 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.359919    2549 scope.go:117] "RemoveContainer" containerID="3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2"
May 25 13:44:40.360213 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:40.360195    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2\": container with ID starting with 3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2 not found: ID does not exist" containerID="3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2"
May 25 13:44:40.360313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.360221    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2"} err="failed to get container status \"3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2\": rpc error: code = NotFound desc = could not find container \"3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2\": container with ID starting with 3581b2c26551aceb5a384dac99793ca0232a2cd9a9f10e0108f34a776ac04df2 not found: ID does not exist"
May 25 13:44:40.360313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.360238    2549 scope.go:117] "RemoveContainer" containerID="e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c"
May 25 13:44:40.360498 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:40.360479    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c\": container with ID starting with e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c not found: ID does not exist" containerID="e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c"
May 25 13:44:40.360551 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.360503    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c"} err="failed to get container status \"e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c\": rpc error: code = NotFound desc = could not find container \"e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c\": container with ID starting with e7a33fd76aba3bc616db65e5badc06b8ed92d66a85aed801f0ec74a4abb0856c not found: ID does not exist"
May 25 13:44:40.360551 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.360518    2549 scope.go:117] "RemoveContainer" containerID="965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9"
May 25 13:44:40.360727 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:40.360708    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9\": container with ID starting with 965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9 not found: ID does not exist" containerID="965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9"
May 25 13:44:40.360786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.360734    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9"} err="failed to get container status \"965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9\": rpc error: code = NotFound desc = could not find container \"965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9\": container with ID starting with 965edeae5941a73d47583fc960a3d8e3d8bbf3c5838daaaf8ae4df38a8eee5e9 not found: ID does not exist"
May 25 13:44:40.361047 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.361028    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"]
May 25 13:44:40.366164 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.366142    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d79fbd-predictor-64c4c46948-xhdsc"]
May 25 13:44:40.496129 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:40.496096    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82597652-966b-4713-a591-e3d20a02c6f8" path="/var/lib/kubelet/pods/82597652-966b-4713-a591-e3d20a02c6f8/volumes"
May 25 13:44:42.344483 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:42.344456    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7_b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49/storage-initializer/0.log"
May 25 13:44:42.344848 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:42.344494    2549 generic.go:358] "Generic (PLEG): container finished" podID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerID="db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5" exitCode=1
May 25 13:44:42.344848 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:42.344521    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7" event={"ID":"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49","Type":"ContainerDied","Data":"db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5"}
May 25 13:44:43.351889 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:43.351859    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7_b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49/storage-initializer/0.log"
May 25 13:44:43.352239 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:43.351931    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7" event={"ID":"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49","Type":"ContainerStarted","Data":"8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9"}
May 25 13:44:45.488179 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.488148    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"]
May 25 13:44:45.488648 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.488464    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7" podUID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerName="storage-initializer" containerID="cri-o://8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9" gracePeriod=30
May 25 13:44:45.650651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650611    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"]
May 25 13:44:45.650893 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650878    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kube-rbac-proxy"
May 25 13:44:45.650944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650893    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kube-rbac-proxy"
May 25 13:44:45.650944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650903    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" containerName="storage-initializer"
May 25 13:44:45.650944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650908    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" containerName="storage-initializer"
May 25 13:44:45.650944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650919    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" containerName="storage-initializer"
May 25 13:44:45.650944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650924    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" containerName="storage-initializer"
May 25 13:44:45.650944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650934    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="storage-initializer"
May 25 13:44:45.650944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650938    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="storage-initializer"
May 25 13:44:45.650944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650947    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container"
May 25 13:44:45.651205 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650952    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container"
May 25 13:44:45.651205 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650990    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" containerName="storage-initializer"
May 25 13:44:45.651205 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.650999    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kserve-container"
May 25 13:44:45.651205 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.651005    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="82597652-966b-4713-a591-e3d20a02c6f8" containerName="kube-rbac-proxy"
May 25 13:44:45.651205 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.651081    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="35b62580-52fa-4038-92a8-9d43dcb1b5f8" containerName="storage-initializer"
May 25 13:44:45.653945 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.653929    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.656156 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.656131    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
May 25 13:44:45.656156 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.656140    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
May 25 13:44:45.656363 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.656178    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zwsdr\""
May 25 13:44:45.662398 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.662374    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"]
May 25 13:44:45.702956 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.702923    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90afbe29-6ec4-48cc-8709-9bf8609a2378-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.703150 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.702982    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tvb\" (UniqueName: \"kubernetes.io/projected/90afbe29-6ec4-48cc-8709-9bf8609a2378-kube-api-access-w8tvb\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.703150 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.703072    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.703150 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.703133    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90afbe29-6ec4-48cc-8709-9bf8609a2378-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.804138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.804049    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90afbe29-6ec4-48cc-8709-9bf8609a2378-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.804138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.804099    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90afbe29-6ec4-48cc-8709-9bf8609a2378-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.804138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.804133    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tvb\" (UniqueName: \"kubernetes.io/projected/90afbe29-6ec4-48cc-8709-9bf8609a2378-kube-api-access-w8tvb\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.804398 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.804159    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.804398 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:45.804260    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
May 25 13:44:45.804398 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:45.804348    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls podName:90afbe29-6ec4-48cc-8709-9bf8609a2378 nodeName:}" failed. No retries permitted until 2026-05-25 13:44:46.304331491 +0000 UTC m=+2436.343088592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls") pod "isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" (UID: "90afbe29-6ec4-48cc-8709-9bf8609a2378") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
May 25 13:44:45.804657 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.804635    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90afbe29-6ec4-48cc-8709-9bf8609a2378-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.804855 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.804836    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90afbe29-6ec4-48cc-8709-9bf8609a2378-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:45.812635 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:45.812612    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tvb\" (UniqueName: \"kubernetes.io/projected/90afbe29-6ec4-48cc-8709-9bf8609a2378-kube-api-access-w8tvb\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:46.309087 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:46.309037    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:46.311636 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:46.311617    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:46.563891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:46.563795    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:44:46.686370 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:46.686341    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"]
May 25 13:44:46.688934 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:44:46.688906    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90afbe29_6ec4_48cc_8709_9bf8609a2378.slice/crio-3bd7c5be191177756210c2f7cb063bfcec17c97de769aa14b5ced0940c04c796 WatchSource:0}: Error finding container 3bd7c5be191177756210c2f7cb063bfcec17c97de769aa14b5ced0940c04c796: Status 404 returned error can't find the container with id 3bd7c5be191177756210c2f7cb063bfcec17c97de769aa14b5ced0940c04c796
May 25 13:44:47.364289 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:47.364232    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" event={"ID":"90afbe29-6ec4-48cc-8709-9bf8609a2378","Type":"ContainerStarted","Data":"09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66"}
May 25 13:44:47.364289 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:47.364290    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" event={"ID":"90afbe29-6ec4-48cc-8709-9bf8609a2378","Type":"ContainerStarted","Data":"3bd7c5be191177756210c2f7cb063bfcec17c97de769aa14b5ced0940c04c796"}
May 25 13:44:48.931701 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:48.931676    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7_b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49/storage-initializer/1.log"
May 25 13:44:48.932071 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:48.932056    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7_b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49/storage-initializer/0.log"
May 25 13:44:48.932135 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:48.932125    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:49.029654 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.029548    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\") pod \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") "
May 25 13:44:49.029654 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.029593    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls\") pod \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") "
May 25 13:44:49.029654 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.029617    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kserve-provision-location\") pod \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") "
May 25 13:44:49.029939 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.029661    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lswfr\" (UniqueName: \"kubernetes.io/projected/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kube-api-access-lswfr\") pod \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") "
May 25 13:44:49.029939 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.029680    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-cabundle-cert\") pod \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\" (UID: \"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49\") "
May 25 13:44:49.030048 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.029936    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" (UID: "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:44:49.030091 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.030031    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-isvc-init-fail-fa253b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-fa253b-kube-rbac-proxy-sar-config") pod "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" (UID: "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49"). InnerVolumeSpecName "isvc-init-fail-fa253b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:44:49.030129 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.030093    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" (UID: "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:44:49.031887 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.031865    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" (UID: "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:44:49.032000 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.031887    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kube-api-access-lswfr" (OuterVolumeSpecName: "kube-api-access-lswfr") pod "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" (UID: "b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49"). InnerVolumeSpecName "kube-api-access-lswfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:44:49.130911 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.130872    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:49.130911 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.130907    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:49.130911 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.130917    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lswfr\" (UniqueName: \"kubernetes.io/projected/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-kube-api-access-lswfr\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:49.131137 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.130926    2549 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-cabundle-cert\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:49.131137 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.130935    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49-isvc-init-fail-fa253b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:44:49.370690 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.370658    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7_b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49/storage-initializer/1.log"
May 25 13:44:49.371032 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.371015    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7_b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49/storage-initializer/0.log"
May 25 13:44:49.371099 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.371050    2549 generic.go:358] "Generic (PLEG): container finished" podID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerID="8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9" exitCode=1
May 25 13:44:49.371138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.371103    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7" event={"ID":"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49","Type":"ContainerDied","Data":"8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9"}
May 25 13:44:49.371138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.371126    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7" event={"ID":"b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49","Type":"ContainerDied","Data":"d6c2e358cd20c816134b4ac0e7f472d3c637299ba3fb27a0d9244e1c35d9a41a"}
May 25 13:44:49.371197 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.371141    2549 scope.go:117] "RemoveContainer" containerID="8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9"
May 25 13:44:49.371197 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.371142    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"
May 25 13:44:49.379456 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.379431    2549 scope.go:117] "RemoveContainer" containerID="db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5"
May 25 13:44:49.386790 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.386771    2549 scope.go:117] "RemoveContainer" containerID="8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9"
May 25 13:44:49.387037 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:49.387015    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9\": container with ID starting with 8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9 not found: ID does not exist" containerID="8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9"
May 25 13:44:49.387104 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.387043    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9"} err="failed to get container status \"8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9\": rpc error: code = NotFound desc = could not find container \"8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9\": container with ID starting with 8724040fa5a0ce248416dee089bf165b575480adfbe2d9b206f963700bb397a9 not found: ID does not exist"
May 25 13:44:49.387104 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.387058    2549 scope.go:117] "RemoveContainer" containerID="db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5"
May 25 13:44:49.387259 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:44:49.387245    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5\": container with ID starting with db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5 not found: ID does not exist" containerID="db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5"
May 25 13:44:49.387336 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.387263    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5"} err="failed to get container status \"db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5\": rpc error: code = NotFound desc = could not find container \"db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5\": container with ID starting with db04ef9eb9c04fe1dd03c95a398bdfe9f946293ec79e332a9449d209885f93e5 not found: ID does not exist"
May 25 13:44:49.407116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.407083    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"]
May 25 13:44:49.409003 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:49.408976    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fa253b-predictor-7b75cc49bc-m4jt7"]
May 25 13:44:50.376127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:50.376095    2549 generic.go:358] "Generic (PLEG): container finished" podID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerID="09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66" exitCode=0
May 25 13:44:50.376615 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:50.376148    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" event={"ID":"90afbe29-6ec4-48cc-8709-9bf8609a2378","Type":"ContainerDied","Data":"09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66"}
May 25 13:44:50.496355 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:44:50.496323    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" path="/var/lib/kubelet/pods/b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49/volumes"
May 25 13:45:11.450797 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:11.450709    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" event={"ID":"90afbe29-6ec4-48cc-8709-9bf8609a2378","Type":"ContainerStarted","Data":"f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0"}
May 25 13:45:11.450797 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:11.450756    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" event={"ID":"90afbe29-6ec4-48cc-8709-9bf8609a2378","Type":"ContainerStarted","Data":"1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2"}
May 25 13:45:11.451228 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:11.451101    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:45:11.472835 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:11.472781    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podStartSLOduration=5.9626482880000005 podStartE2EDuration="26.472766641s" podCreationTimestamp="2026-05-25 13:44:45 +0000 UTC" firstStartedPulling="2026-05-25 13:44:50.377319404 +0000 UTC m=+2440.416076505" lastFinishedPulling="2026-05-25 13:45:10.887437743 +0000 UTC m=+2460.926194858" observedRunningTime="2026-05-25 13:45:11.470915075 +0000 UTC m=+2461.509672197" watchObservedRunningTime="2026-05-25 13:45:11.472766641 +0000 UTC m=+2461.511523822"
May 25 13:45:12.453718 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:12.453677    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:45:12.454964 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:12.454931    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:45:13.456622 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:13.456579    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:45:18.460941 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:18.460911    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:45:18.461571 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:18.461543    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:45:28.461717 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:28.461680    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:45:38.461575 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:38.461528    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:45:48.462002 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:48.461963    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:45:58.461899 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:45:58.461857    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:46:08.462378 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:08.462332    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:46:18.462379 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:18.462338    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:46:28.462451 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:28.462416    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:46:35.809700 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.809666    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"]
May 25 13:46:35.810171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.809961    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" containerID="cri-o://1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2" gracePeriod=30
May 25 13:46:35.810171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.810035    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kube-rbac-proxy" containerID="cri-o://f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0" gracePeriod=30
May 25 13:46:35.907625 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.907588    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"]
May 25 13:46:35.907883 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.907871    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerName="storage-initializer"
May 25 13:46:35.907944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.907885    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerName="storage-initializer"
May 25 13:46:35.907944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.907901    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerName="storage-initializer"
May 25 13:46:35.907944 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.907907    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerName="storage-initializer"
May 25 13:46:35.908047 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.907957    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerName="storage-initializer"
May 25 13:46:35.908047 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.907965    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="b3dcc9f8-8cc8-4cac-a0b0-b879fc9bad49" containerName="storage-initializer"
May 25 13:46:35.911219 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.911201    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:35.913469 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.913447    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
May 25 13:46:35.913570 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.913456    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
May 25 13:46:35.922000 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:35.921973    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"]
May 25 13:46:36.066561 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.066447    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b69d26e-c41e-4652-bdde-7504c70ee7a6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.066561 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.066503    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.066561 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.066529    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kft\" (UniqueName: \"kubernetes.io/projected/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kube-api-access-86kft\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.066791 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.066609    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b69d26e-c41e-4652-bdde-7504c70ee7a6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.167805 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.167760    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.167805 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.167808    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-86kft\" (UniqueName: \"kubernetes.io/projected/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kube-api-access-86kft\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.168068 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.167857    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b69d26e-c41e-4652-bdde-7504c70ee7a6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.168068 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.167881    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b69d26e-c41e-4652-bdde-7504c70ee7a6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.168192 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.168171    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.168494 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.168475    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b69d26e-c41e-4652-bdde-7504c70ee7a6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.170457 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.170440    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b69d26e-c41e-4652-bdde-7504c70ee7a6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.176051 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.176023    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kft\" (UniqueName: \"kubernetes.io/projected/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kube-api-access-86kft\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.222506 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.222470    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:36.351177 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.351144    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"]
May 25 13:46:36.353839 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:46:36.353808    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b69d26e_c41e_4652_bdde_7504c70ee7a6.slice/crio-1bd7d0599eb7f80358729af71229781d1aeb81bf5e2d2307084def31e8780d64 WatchSource:0}: Error finding container 1bd7d0599eb7f80358729af71229781d1aeb81bf5e2d2307084def31e8780d64: Status 404 returned error can't find the container with id 1bd7d0599eb7f80358729af71229781d1aeb81bf5e2d2307084def31e8780d64
May 25 13:46:36.355659 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.355643    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:46:36.690737 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.690704    2549 generic.go:358] "Generic (PLEG): container finished" podID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerID="f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0" exitCode=2
May 25 13:46:36.690950 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.690778    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" event={"ID":"90afbe29-6ec4-48cc-8709-9bf8609a2378","Type":"ContainerDied","Data":"f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0"}
May 25 13:46:36.692329 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.692300    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" event={"ID":"8b69d26e-c41e-4652-bdde-7504c70ee7a6","Type":"ContainerStarted","Data":"0decb99efd94c1a3187cf84d7dca4ce4c77a82b9118686a0ff0bc936f8103ce7"}
May 25 13:46:36.692476 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:36.692338    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" event={"ID":"8b69d26e-c41e-4652-bdde-7504c70ee7a6","Type":"ContainerStarted","Data":"1bd7d0599eb7f80358729af71229781d1aeb81bf5e2d2307084def31e8780d64"}
May 25 13:46:38.456900 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:38.456854    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 25 13:46:38.461821 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:38.461766    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 13:46:40.704425 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:40.704394    2549 generic.go:358] "Generic (PLEG): container finished" podID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerID="0decb99efd94c1a3187cf84d7dca4ce4c77a82b9118686a0ff0bc936f8103ce7" exitCode=0
May 25 13:46:40.704714 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:40.704463    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" event={"ID":"8b69d26e-c41e-4652-bdde-7504c70ee7a6","Type":"ContainerDied","Data":"0decb99efd94c1a3187cf84d7dca4ce4c77a82b9118686a0ff0bc936f8103ce7"}
May 25 13:46:40.870086 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:40.870062    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:46:41.004755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.004659    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls\") pod \"90afbe29-6ec4-48cc-8709-9bf8609a2378\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") "
May 25 13:46:41.004755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.004712    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8tvb\" (UniqueName: \"kubernetes.io/projected/90afbe29-6ec4-48cc-8709-9bf8609a2378-kube-api-access-w8tvb\") pod \"90afbe29-6ec4-48cc-8709-9bf8609a2378\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") "
May 25 13:46:41.004987 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.004757    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90afbe29-6ec4-48cc-8709-9bf8609a2378-kserve-provision-location\") pod \"90afbe29-6ec4-48cc-8709-9bf8609a2378\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") "
May 25 13:46:41.004987 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.004786    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90afbe29-6ec4-48cc-8709-9bf8609a2378-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"90afbe29-6ec4-48cc-8709-9bf8609a2378\" (UID: \"90afbe29-6ec4-48cc-8709-9bf8609a2378\") "
May 25 13:46:41.005134 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.005104    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90afbe29-6ec4-48cc-8709-9bf8609a2378-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90afbe29-6ec4-48cc-8709-9bf8609a2378" (UID: "90afbe29-6ec4-48cc-8709-9bf8609a2378"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:46:41.005246 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.005209    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90afbe29-6ec4-48cc-8709-9bf8609a2378-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "90afbe29-6ec4-48cc-8709-9bf8609a2378" (UID: "90afbe29-6ec4-48cc-8709-9bf8609a2378"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:46:41.006893 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.006871    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90afbe29-6ec4-48cc-8709-9bf8609a2378" (UID: "90afbe29-6ec4-48cc-8709-9bf8609a2378"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:46:41.006971 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.006936    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90afbe29-6ec4-48cc-8709-9bf8609a2378-kube-api-access-w8tvb" (OuterVolumeSpecName: "kube-api-access-w8tvb") pod "90afbe29-6ec4-48cc-8709-9bf8609a2378" (UID: "90afbe29-6ec4-48cc-8709-9bf8609a2378"). InnerVolumeSpecName "kube-api-access-w8tvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:46:41.106182 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.106142    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90afbe29-6ec4-48cc-8709-9bf8609a2378-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:46:41.106182 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.106174    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8tvb\" (UniqueName: \"kubernetes.io/projected/90afbe29-6ec4-48cc-8709-9bf8609a2378-kube-api-access-w8tvb\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:46:41.106182 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.106185    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90afbe29-6ec4-48cc-8709-9bf8609a2378-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:46:41.106431 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.106194    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90afbe29-6ec4-48cc-8709-9bf8609a2378-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:46:41.708702 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.708667    2549 generic.go:358] "Generic (PLEG): container finished" podID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerID="1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2" exitCode=0
May 25 13:46:41.709127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.708732    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" event={"ID":"90afbe29-6ec4-48cc-8709-9bf8609a2378","Type":"ContainerDied","Data":"1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2"}
May 25 13:46:41.709127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.708757    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k" event={"ID":"90afbe29-6ec4-48cc-8709-9bf8609a2378","Type":"ContainerDied","Data":"3bd7c5be191177756210c2f7cb063bfcec17c97de769aa14b5ced0940c04c796"}
May 25 13:46:41.709127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.708758    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"
May 25 13:46:41.709127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.708773    2549 scope.go:117] "RemoveContainer" containerID="f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0"
May 25 13:46:41.710995 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.710967    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" event={"ID":"8b69d26e-c41e-4652-bdde-7504c70ee7a6","Type":"ContainerStarted","Data":"409cfe1501f3666bbd41d0ed4af0c988dbd7127db466fd715c9e1fb557552d34"}
May 25 13:46:41.711102 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.711010    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" event={"ID":"8b69d26e-c41e-4652-bdde-7504c70ee7a6","Type":"ContainerStarted","Data":"ab9995905f0efee3701dbb12af6173da2c3ec7b38824b0d96eebfea48c508b57"}
May 25 13:46:41.711256 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.711235    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:41.711356 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.711289    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:41.712982 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.712950    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:46:41.716731 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.716714    2549 scope.go:117] "RemoveContainer" containerID="1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2"
May 25 13:46:41.723780 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.723753    2549 scope.go:117] "RemoveContainer" containerID="09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66"
May 25 13:46:41.731014 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.730995    2549 scope.go:117] "RemoveContainer" containerID="f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0"
May 25 13:46:41.731304 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:46:41.731259    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0\": container with ID starting with f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0 not found: ID does not exist" containerID="f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0"
May 25 13:46:41.731411 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.731315    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0"} err="failed to get container status \"f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0\": rpc error: code = NotFound desc = could not find container \"f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0\": container with ID starting with f315b2e0d10ff1b25342cc6cb8523a664041ec85cecaf15f0874cd2938ca55b0 not found: ID does not exist"
May 25 13:46:41.731411 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.731340    2549 scope.go:117] "RemoveContainer" containerID="1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2"
May 25 13:46:41.731411 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.731360    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podStartSLOduration=6.731349109 podStartE2EDuration="6.731349109s" podCreationTimestamp="2026-05-25 13:46:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:46:41.730428629 +0000 UTC m=+2551.769185755" watchObservedRunningTime="2026-05-25 13:46:41.731349109 +0000 UTC m=+2551.770106233"
May 25 13:46:41.731625 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:46:41.731608    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2\": container with ID starting with 1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2 not found: ID does not exist" containerID="1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2"
May 25 13:46:41.731665 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.731631    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2"} err="failed to get container status \"1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2\": rpc error: code = NotFound desc = could not find container \"1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2\": container with ID starting with 1ff00f4a5a2515b6e207e551f8dd125b73766afbf9faa3667809372c1eefbce2 not found: ID does not exist"
May 25 13:46:41.731665 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.731647    2549 scope.go:117] "RemoveContainer" containerID="09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66"
May 25 13:46:41.731886 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:46:41.731867    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66\": container with ID starting with 09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66 not found: ID does not exist" containerID="09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66"
May 25 13:46:41.731942 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.731890    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66"} err="failed to get container status \"09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66\": rpc error: code = NotFound desc = could not find container \"09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66\": container with ID starting with 09ea441fa2114b22c1da50001ccbebb6d8ae8ec6cb18b7df48223eb22eb9cb66 not found: ID does not exist"
May 25 13:46:41.743786 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.743759    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"]
May 25 13:46:41.747946 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:41.747924    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-vzx5k"]
May 25 13:46:42.495533 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:42.495498    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" path="/var/lib/kubelet/pods/90afbe29-6ec4-48cc-8709-9bf8609a2378/volumes"
May 25 13:46:42.714598 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:42.714559    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:46:47.719391 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:47.719357    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:46:47.719887 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:47.719861    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:46:57.720464 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:46:57.720417    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:47:07.720194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:47:07.720154    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:47:17.719893 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:47:17.719851    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:47:27.720643 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:47:27.720552    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:47:37.720463 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:47:37.720419    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:47:47.720484 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:47:47.720444    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:47:57.720446 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:47:57.720413    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:48:06.041730 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.041694    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"]
May 25 13:48:06.042162 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.042008    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" containerID="cri-o://ab9995905f0efee3701dbb12af6173da2c3ec7b38824b0d96eebfea48c508b57" gracePeriod=30
May 25 13:48:06.042162 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.042097    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kube-rbac-proxy" containerID="cri-o://409cfe1501f3666bbd41d0ed4af0c988dbd7127db466fd715c9e1fb557552d34" gracePeriod=30
May 25 13:48:06.141885 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.141839    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"]
May 25 13:48:06.142170 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.142155    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container"
May 25 13:48:06.142215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.142174    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container"
May 25 13:48:06.142215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.142191    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="storage-initializer"
May 25 13:48:06.142215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.142197    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="storage-initializer"
May 25 13:48:06.142215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.142204    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kube-rbac-proxy"
May 25 13:48:06.142215 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.142210    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kube-rbac-proxy"
May 25 13:48:06.142392 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.142259    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kserve-container"
May 25 13:48:06.142392 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.142283    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="90afbe29-6ec4-48cc-8709-9bf8609a2378" containerName="kube-rbac-proxy"
May 25 13:48:06.145276 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.145250    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.147574 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.147544    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
May 25 13:48:06.147704 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.147553    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
May 25 13:48:06.154799 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.154774    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"]
May 25 13:48:06.248990 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.248950    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a5a54d-757c-4d85-be37-74c4b56a24ce-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.249173 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.248998    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a5a54d-757c-4d85-be37-74c4b56a24ce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.249173 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.249057    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a5a54d-757c-4d85-be37-74c4b56a24ce-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.249173 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.249129    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcql\" (UniqueName: \"kubernetes.io/projected/42a5a54d-757c-4d85-be37-74c4b56a24ce-kube-api-access-wfcql\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.349602 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.349514    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a5a54d-757c-4d85-be37-74c4b56a24ce-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.349602 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.349567    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcql\" (UniqueName: \"kubernetes.io/projected/42a5a54d-757c-4d85-be37-74c4b56a24ce-kube-api-access-wfcql\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.349808 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.349636    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a5a54d-757c-4d85-be37-74c4b56a24ce-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.349808 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.349674    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a5a54d-757c-4d85-be37-74c4b56a24ce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.350059 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.350038    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a5a54d-757c-4d85-be37-74c4b56a24ce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.350301 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.350263    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a5a54d-757c-4d85-be37-74c4b56a24ce-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.352152 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.352132    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a5a54d-757c-4d85-be37-74c4b56a24ce-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.357556 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.357535    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcql\" (UniqueName: \"kubernetes.io/projected/42a5a54d-757c-4d85-be37-74c4b56a24ce-kube-api-access-wfcql\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.456646 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.456605    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:06.590368 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.590330    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"]
May 25 13:48:06.593545 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:48:06.593513    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a5a54d_757c_4d85_be37_74c4b56a24ce.slice/crio-c9f43b522b782442ddf25e5361f03b80fbf47ee85684feb09f7bfbafe7cf25d6 WatchSource:0}: Error finding container c9f43b522b782442ddf25e5361f03b80fbf47ee85684feb09f7bfbafe7cf25d6: Status 404 returned error can't find the container with id c9f43b522b782442ddf25e5361f03b80fbf47ee85684feb09f7bfbafe7cf25d6
May 25 13:48:06.944734 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.944698    2549 generic.go:358] "Generic (PLEG): container finished" podID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerID="409cfe1501f3666bbd41d0ed4af0c988dbd7127db466fd715c9e1fb557552d34" exitCode=2
May 25 13:48:06.944927 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.944782    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" event={"ID":"8b69d26e-c41e-4652-bdde-7504c70ee7a6","Type":"ContainerDied","Data":"409cfe1501f3666bbd41d0ed4af0c988dbd7127db466fd715c9e1fb557552d34"}
May 25 13:48:06.946118 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.946083    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" event={"ID":"42a5a54d-757c-4d85-be37-74c4b56a24ce","Type":"ContainerStarted","Data":"0cea06b5881c5f7980f7e5a81b18fd931ae747d768cb566aca98398c458760a1"}
May 25 13:48:06.946118 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:06.946116    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" event={"ID":"42a5a54d-757c-4d85-be37-74c4b56a24ce","Type":"ContainerStarted","Data":"c9f43b522b782442ddf25e5361f03b80fbf47ee85684feb09f7bfbafe7cf25d6"}
May 25 13:48:07.715588 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:07.715538    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
May 25 13:48:07.719921 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:07.719897    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 13:48:10.961639 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:10.961601    2549 generic.go:358] "Generic (PLEG): container finished" podID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerID="0cea06b5881c5f7980f7e5a81b18fd931ae747d768cb566aca98398c458760a1" exitCode=0
May 25 13:48:10.961970 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:10.961637    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" event={"ID":"42a5a54d-757c-4d85-be37-74c4b56a24ce","Type":"ContainerDied","Data":"0cea06b5881c5f7980f7e5a81b18fd931ae747d768cb566aca98398c458760a1"}
May 25 13:48:10.963794 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:10.963774    2549 generic.go:358] "Generic (PLEG): container finished" podID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerID="ab9995905f0efee3701dbb12af6173da2c3ec7b38824b0d96eebfea48c508b57" exitCode=0
May 25 13:48:10.963891 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:10.963838    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" event={"ID":"8b69d26e-c41e-4652-bdde-7504c70ee7a6","Type":"ContainerDied","Data":"ab9995905f0efee3701dbb12af6173da2c3ec7b38824b0d96eebfea48c508b57"}
May 25 13:48:11.111039 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.111016    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:48:11.190463 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.190363    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b69d26e-c41e-4652-bdde-7504c70ee7a6-proxy-tls\") pod \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") "
May 25 13:48:11.190463 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.190431    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kserve-provision-location\") pod \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") "
May 25 13:48:11.190753 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.190483    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b69d26e-c41e-4652-bdde-7504c70ee7a6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") "
May 25 13:48:11.190753 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.190527    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86kft\" (UniqueName: \"kubernetes.io/projected/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kube-api-access-86kft\") pod \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\" (UID: \"8b69d26e-c41e-4652-bdde-7504c70ee7a6\") "
May 25 13:48:11.190875 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.190845    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b69d26e-c41e-4652-bdde-7504c70ee7a6" (UID: "8b69d26e-c41e-4652-bdde-7504c70ee7a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:48:11.190936 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.190876    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b69d26e-c41e-4652-bdde-7504c70ee7a6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "8b69d26e-c41e-4652-bdde-7504c70ee7a6" (UID: "8b69d26e-c41e-4652-bdde-7504c70ee7a6"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:48:11.192842 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.192808    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kube-api-access-86kft" (OuterVolumeSpecName: "kube-api-access-86kft") pod "8b69d26e-c41e-4652-bdde-7504c70ee7a6" (UID: "8b69d26e-c41e-4652-bdde-7504c70ee7a6"). InnerVolumeSpecName "kube-api-access-86kft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:48:11.193139 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.193116    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b69d26e-c41e-4652-bdde-7504c70ee7a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b69d26e-c41e-4652-bdde-7504c70ee7a6" (UID: "8b69d26e-c41e-4652-bdde-7504c70ee7a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:48:11.291825 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.291778    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b69d26e-c41e-4652-bdde-7504c70ee7a6-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:48:11.291825 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.291823    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:48:11.292041 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.291838    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b69d26e-c41e-4652-bdde-7504c70ee7a6-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:48:11.292041 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.291853    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86kft\" (UniqueName: \"kubernetes.io/projected/8b69d26e-c41e-4652-bdde-7504c70ee7a6-kube-api-access-86kft\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:48:11.969003 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.968971    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" event={"ID":"42a5a54d-757c-4d85-be37-74c4b56a24ce","Type":"ContainerStarted","Data":"f85523343f708c65df1b46b4ad4c46e415b7905bcc69a93d279495d9ce43ba07"}
May 25 13:48:11.969623 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.969015    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" event={"ID":"42a5a54d-757c-4d85-be37-74c4b56a24ce","Type":"ContainerStarted","Data":"9ad98019f03c8825ec03f52f2d603295b523d1b46765f6b6d3fefd19afacc885"}
May 25 13:48:11.969623 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.969308    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:11.969623 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.969445    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:11.970605 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.970574    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:48:11.970841 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.970821    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf" event={"ID":"8b69d26e-c41e-4652-bdde-7504c70ee7a6","Type":"ContainerDied","Data":"1bd7d0599eb7f80358729af71229781d1aeb81bf5e2d2307084def31e8780d64"}
May 25 13:48:11.970932 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.970863    2549 scope.go:117] "RemoveContainer" containerID="409cfe1501f3666bbd41d0ed4af0c988dbd7127db466fd715c9e1fb557552d34"
May 25 13:48:11.970932 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.970879    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"
May 25 13:48:11.979096 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.979080    2549 scope.go:117] "RemoveContainer" containerID="ab9995905f0efee3701dbb12af6173da2c3ec7b38824b0d96eebfea48c508b57"
May 25 13:48:11.986166 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.986149    2549 scope.go:117] "RemoveContainer" containerID="0decb99efd94c1a3187cf84d7dca4ce4c77a82b9118686a0ff0bc936f8103ce7"
May 25 13:48:11.986563 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.986512    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podStartSLOduration=5.986500543 podStartE2EDuration="5.986500543s" podCreationTimestamp="2026-05-25 13:48:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:48:11.98584462 +0000 UTC m=+2642.024601743" watchObservedRunningTime="2026-05-25 13:48:11.986500543 +0000 UTC m=+2642.025257666"
May 25 13:48:11.997939 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:11.997907    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"]
May 25 13:48:12.001597 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:12.001574    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-v82xf"]
May 25 13:48:12.495700 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:12.495666    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" path="/var/lib/kubelet/pods/8b69d26e-c41e-4652-bdde-7504c70ee7a6/volumes"
May 25 13:48:12.974866 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:12.974832    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:48:17.980226 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:17.980196    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:48:17.980743 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:17.980717    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:48:27.981431 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:27.981357    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:48:37.981107 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:37.981058    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:48:47.980669 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:47.980625    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:48:57.981058 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:48:57.980967    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:49:07.981084 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:07.981034    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:49:11.157729 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:11.157692    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:49:11.162063 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:11.162040    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:49:17.980719 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:17.980674    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:49:27.981494 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:27.981454    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:49:36.264416 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.264378    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"]
May 25 13:49:36.264925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.264834    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kube-rbac-proxy" containerID="cri-o://f85523343f708c65df1b46b4ad4c46e415b7905bcc69a93d279495d9ce43ba07" gracePeriod=30
May 25 13:49:36.265005 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.264818    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" containerID="cri-o://9ad98019f03c8825ec03f52f2d603295b523d1b46765f6b6d3fefd19afacc885" gracePeriod=30
May 25 13:49:36.376906 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.376872    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"]
May 25 13:49:36.377170 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.377154    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container"
May 25 13:49:36.377231 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.377174    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container"
May 25 13:49:36.377231 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.377194    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="storage-initializer"
May 25 13:49:36.377231 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.377202    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="storage-initializer"
May 25 13:49:36.377231 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.377223    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kube-rbac-proxy"
May 25 13:49:36.377409 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.377233    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kube-rbac-proxy"
May 25 13:49:36.377409 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.377308    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kube-rbac-proxy"
May 25 13:49:36.377409 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.377321    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="8b69d26e-c41e-4652-bdde-7504c70ee7a6" containerName="kserve-container"
May 25 13:49:36.381593 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.381569    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.384094 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.384067    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
May 25 13:49:36.384197 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.384067    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
May 25 13:49:36.391893 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.391851    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"]
May 25 13:49:36.518380 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.518288    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9374bef7-0f72-4a04-8c2a-67036710c93f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.518380 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.518330    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdqp\" (UniqueName: \"kubernetes.io/projected/9374bef7-0f72-4a04-8c2a-67036710c93f-kube-api-access-qmdqp\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.518613 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.518395    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9374bef7-0f72-4a04-8c2a-67036710c93f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.518613 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.518467    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9374bef7-0f72-4a04-8c2a-67036710c93f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.619863 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.619824    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9374bef7-0f72-4a04-8c2a-67036710c93f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.620037 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.619871    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9374bef7-0f72-4a04-8c2a-67036710c93f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.620099 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.620077    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9374bef7-0f72-4a04-8c2a-67036710c93f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.620137 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.620126    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdqp\" (UniqueName: \"kubernetes.io/projected/9374bef7-0f72-4a04-8c2a-67036710c93f-kube-api-access-qmdqp\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.620440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.620400    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9374bef7-0f72-4a04-8c2a-67036710c93f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.620608 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.620586    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9374bef7-0f72-4a04-8c2a-67036710c93f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.622496 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.622470    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9374bef7-0f72-4a04-8c2a-67036710c93f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.628225 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.628196    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdqp\" (UniqueName: \"kubernetes.io/projected/9374bef7-0f72-4a04-8c2a-67036710c93f-kube-api-access-qmdqp\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.691908 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.691871    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:36.821217 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:36.821024    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"]
May 25 13:49:36.824013 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:49:36.823984    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9374bef7_0f72_4a04_8c2a_67036710c93f.slice/crio-004445c0b6d4f3a26003bf8a0b4f4b2c6777b0283942ae61592511a3b7b9027f WatchSource:0}: Error finding container 004445c0b6d4f3a26003bf8a0b4f4b2c6777b0283942ae61592511a3b7b9027f: Status 404 returned error can't find the container with id 004445c0b6d4f3a26003bf8a0b4f4b2c6777b0283942ae61592511a3b7b9027f
May 25 13:49:37.203998 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:37.203954    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" event={"ID":"9374bef7-0f72-4a04-8c2a-67036710c93f","Type":"ContainerStarted","Data":"d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93"}
May 25 13:49:37.203998 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:37.203998    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" event={"ID":"9374bef7-0f72-4a04-8c2a-67036710c93f","Type":"ContainerStarted","Data":"004445c0b6d4f3a26003bf8a0b4f4b2c6777b0283942ae61592511a3b7b9027f"}
May 25 13:49:37.205976 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:37.205946    2549 generic.go:358] "Generic (PLEG): container finished" podID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerID="f85523343f708c65df1b46b4ad4c46e415b7905bcc69a93d279495d9ce43ba07" exitCode=2
May 25 13:49:37.206092 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:37.205979    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" event={"ID":"42a5a54d-757c-4d85-be37-74c4b56a24ce","Type":"ContainerDied","Data":"f85523343f708c65df1b46b4ad4c46e415b7905bcc69a93d279495d9ce43ba07"}
May 25 13:49:37.975334 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:37.975265    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
May 25 13:49:37.981636 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:37.981598    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 13:49:41.218483 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:41.218443    2549 generic.go:358] "Generic (PLEG): container finished" podID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerID="d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93" exitCode=0
May 25 13:49:41.218992 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:41.218501    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" event={"ID":"9374bef7-0f72-4a04-8c2a-67036710c93f","Type":"ContainerDied","Data":"d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93"}
May 25 13:49:42.224176 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.224139    2549 generic.go:358] "Generic (PLEG): container finished" podID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerID="9ad98019f03c8825ec03f52f2d603295b523d1b46765f6b6d3fefd19afacc885" exitCode=0
May 25 13:49:42.224631 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.224211    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" event={"ID":"42a5a54d-757c-4d85-be37-74c4b56a24ce","Type":"ContainerDied","Data":"9ad98019f03c8825ec03f52f2d603295b523d1b46765f6b6d3fefd19afacc885"}
May 25 13:49:42.226082 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.226058    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" event={"ID":"9374bef7-0f72-4a04-8c2a-67036710c93f","Type":"ContainerStarted","Data":"cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58"}
May 25 13:49:42.226204 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.226090    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" event={"ID":"9374bef7-0f72-4a04-8c2a-67036710c93f","Type":"ContainerStarted","Data":"63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f"}
May 25 13:49:42.226420 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.226404    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:42.243852 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.243803    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" podStartSLOduration=6.243790544 podStartE2EDuration="6.243790544s" podCreationTimestamp="2026-05-25 13:49:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:49:42.242842626 +0000 UTC m=+2732.281599853" watchObservedRunningTime="2026-05-25 13:49:42.243790544 +0000 UTC m=+2732.282547667"
May 25 13:49:42.305895 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.305870    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:49:42.368122 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.368079    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a5a54d-757c-4d85-be37-74c4b56a24ce-proxy-tls\") pod \"42a5a54d-757c-4d85-be37-74c4b56a24ce\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") "
May 25 13:49:42.368328 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.368152    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a5a54d-757c-4d85-be37-74c4b56a24ce-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"42a5a54d-757c-4d85-be37-74c4b56a24ce\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") "
May 25 13:49:42.368328 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.368208    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a5a54d-757c-4d85-be37-74c4b56a24ce-kserve-provision-location\") pod \"42a5a54d-757c-4d85-be37-74c4b56a24ce\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") "
May 25 13:49:42.368328 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.368253    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfcql\" (UniqueName: \"kubernetes.io/projected/42a5a54d-757c-4d85-be37-74c4b56a24ce-kube-api-access-wfcql\") pod \"42a5a54d-757c-4d85-be37-74c4b56a24ce\" (UID: \"42a5a54d-757c-4d85-be37-74c4b56a24ce\") "
May 25 13:49:42.368635 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.368592    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a5a54d-757c-4d85-be37-74c4b56a24ce-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "42a5a54d-757c-4d85-be37-74c4b56a24ce" (UID: "42a5a54d-757c-4d85-be37-74c4b56a24ce"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:49:42.368729 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.368645    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a5a54d-757c-4d85-be37-74c4b56a24ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42a5a54d-757c-4d85-be37-74c4b56a24ce" (UID: "42a5a54d-757c-4d85-be37-74c4b56a24ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:49:42.370345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.370321    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a5a54d-757c-4d85-be37-74c4b56a24ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42a5a54d-757c-4d85-be37-74c4b56a24ce" (UID: "42a5a54d-757c-4d85-be37-74c4b56a24ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:49:42.370666 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.370643    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a5a54d-757c-4d85-be37-74c4b56a24ce-kube-api-access-wfcql" (OuterVolumeSpecName: "kube-api-access-wfcql") pod "42a5a54d-757c-4d85-be37-74c4b56a24ce" (UID: "42a5a54d-757c-4d85-be37-74c4b56a24ce"). InnerVolumeSpecName "kube-api-access-wfcql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:49:42.469752 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.469661    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a5a54d-757c-4d85-be37-74c4b56a24ce-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:49:42.469752 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.469695    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a5a54d-757c-4d85-be37-74c4b56a24ce-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:49:42.469752 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.469706    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a5a54d-757c-4d85-be37-74c4b56a24ce-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:49:42.469752 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:42.469715    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfcql\" (UniqueName: \"kubernetes.io/projected/42a5a54d-757c-4d85-be37-74c4b56a24ce-kube-api-access-wfcql\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:49:43.230766 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:43.230729    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"
May 25 13:49:43.231222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:43.230721    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp" event={"ID":"42a5a54d-757c-4d85-be37-74c4b56a24ce","Type":"ContainerDied","Data":"c9f43b522b782442ddf25e5361f03b80fbf47ee85684feb09f7bfbafe7cf25d6"}
May 25 13:49:43.231222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:43.230848    2549 scope.go:117] "RemoveContainer" containerID="f85523343f708c65df1b46b4ad4c46e415b7905bcc69a93d279495d9ce43ba07"
May 25 13:49:43.231222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:43.231150    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:49:43.238896 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:43.238880    2549 scope.go:117] "RemoveContainer" containerID="9ad98019f03c8825ec03f52f2d603295b523d1b46765f6b6d3fefd19afacc885"
May 25 13:49:43.245753 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:43.245735    2549 scope.go:117] "RemoveContainer" containerID="0cea06b5881c5f7980f7e5a81b18fd931ae747d768cb566aca98398c458760a1"
May 25 13:49:43.249852 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:43.249831    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"]
May 25 13:49:43.252564 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:43.252542    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-5xqhp"]
May 25 13:49:44.495130 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:44.495096    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" path="/var/lib/kubelet/pods/42a5a54d-757c-4d85-be37-74c4b56a24ce/volumes"
May 25 13:49:49.239857 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:49:49.239830    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:50:19.241451 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:50:19.241357    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 13:50:29.241155 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:50:29.241111    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 13:50:39.240458 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:50:39.240402    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 13:50:49.241373 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:50:49.241327    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 13:50:56.496610 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:50:56.496574    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:51:06.457662 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.457619    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"]
May 25 13:51:06.458171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.458045    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kserve-container" containerID="cri-o://63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f" gracePeriod=30
May 25 13:51:06.458171 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.458084    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kube-rbac-proxy" containerID="cri-o://cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58" gracePeriod=30
May 25 13:51:06.569497 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569463    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"]
May 25 13:51:06.569769 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569756    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kube-rbac-proxy"
May 25 13:51:06.569822 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569771    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kube-rbac-proxy"
May 25 13:51:06.569822 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569791    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container"
May 25 13:51:06.569822 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569796    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container"
May 25 13:51:06.569822 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569805    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="storage-initializer"
May 25 13:51:06.569822 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569811    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="storage-initializer"
May 25 13:51:06.569973 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569856    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kube-rbac-proxy"
May 25 13:51:06.569973 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.569865    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="42a5a54d-757c-4d85-be37-74c4b56a24ce" containerName="kserve-container"
May 25 13:51:06.572754 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.572732    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.574968 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.574945    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
May 25 13:51:06.575130 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.575109    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
May 25 13:51:06.585780 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.585755    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"]
May 25 13:51:06.728658 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.728560    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c42715-506c-4fa3-95e8-89c70c818c94-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.728658 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.728655    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c42715-506c-4fa3-95e8-89c70c818c94-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.728869 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.728682    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkcwd\" (UniqueName: \"kubernetes.io/projected/19c42715-506c-4fa3-95e8-89c70c818c94-kube-api-access-vkcwd\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.728869 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.728705    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c42715-506c-4fa3-95e8-89c70c818c94-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.829535 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.829497    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c42715-506c-4fa3-95e8-89c70c818c94-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.829705 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.829564    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c42715-506c-4fa3-95e8-89c70c818c94-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.829705 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.829588    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vkcwd\" (UniqueName: \"kubernetes.io/projected/19c42715-506c-4fa3-95e8-89c70c818c94-kube-api-access-vkcwd\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.829705 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.829611    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c42715-506c-4fa3-95e8-89c70c818c94-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.830017 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.829989    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c42715-506c-4fa3-95e8-89c70c818c94-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.830372 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.830353    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c42715-506c-4fa3-95e8-89c70c818c94-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.832204 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.832180    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c42715-506c-4fa3-95e8-89c70c818c94-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.838093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.838072    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkcwd\" (UniqueName: \"kubernetes.io/projected/19c42715-506c-4fa3-95e8-89c70c818c94-kube-api-access-vkcwd\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:06.882114 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:06.882076    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:07.014611 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:07.014514    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"]
May 25 13:51:07.017953 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:51:07.017930    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c42715_506c_4fa3_95e8_89c70c818c94.slice/crio-9421300bab8a760948e3201ed35f3e4cd2b98e3f7bfde193bf0931c2c875ed26 WatchSource:0}: Error finding container 9421300bab8a760948e3201ed35f3e4cd2b98e3f7bfde193bf0931c2c875ed26: Status 404 returned error can't find the container with id 9421300bab8a760948e3201ed35f3e4cd2b98e3f7bfde193bf0931c2c875ed26
May 25 13:51:07.467093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:07.467055    2549 generic.go:358] "Generic (PLEG): container finished" podID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerID="cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58" exitCode=2
May 25 13:51:07.467549 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:07.467148    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" event={"ID":"9374bef7-0f72-4a04-8c2a-67036710c93f","Type":"ContainerDied","Data":"cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58"}
May 25 13:51:07.468607 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:07.468580    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" event={"ID":"19c42715-506c-4fa3-95e8-89c70c818c94","Type":"ContainerStarted","Data":"1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2"}
May 25 13:51:07.468719 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:07.468612    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" event={"ID":"19c42715-506c-4fa3-95e8-89c70c818c94","Type":"ContainerStarted","Data":"9421300bab8a760948e3201ed35f3e4cd2b98e3f7bfde193bf0931c2c875ed26"}
May 25 13:51:09.235390 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:09.235340    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
May 25 13:51:11.293764 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.293737    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:51:11.467793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.467755    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9374bef7-0f72-4a04-8c2a-67036710c93f-kserve-provision-location\") pod \"9374bef7-0f72-4a04-8c2a-67036710c93f\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") "
May 25 13:51:11.467793 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.467800    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9374bef7-0f72-4a04-8c2a-67036710c93f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"9374bef7-0f72-4a04-8c2a-67036710c93f\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") "
May 25 13:51:11.468071 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.467843    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9374bef7-0f72-4a04-8c2a-67036710c93f-proxy-tls\") pod \"9374bef7-0f72-4a04-8c2a-67036710c93f\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") "
May 25 13:51:11.468071 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.467898    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmdqp\" (UniqueName: \"kubernetes.io/projected/9374bef7-0f72-4a04-8c2a-67036710c93f-kube-api-access-qmdqp\") pod \"9374bef7-0f72-4a04-8c2a-67036710c93f\" (UID: \"9374bef7-0f72-4a04-8c2a-67036710c93f\") "
May 25 13:51:11.468186 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.468099    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9374bef7-0f72-4a04-8c2a-67036710c93f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9374bef7-0f72-4a04-8c2a-67036710c93f" (UID: "9374bef7-0f72-4a04-8c2a-67036710c93f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:51:11.468186 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.468169    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9374bef7-0f72-4a04-8c2a-67036710c93f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "9374bef7-0f72-4a04-8c2a-67036710c93f" (UID: "9374bef7-0f72-4a04-8c2a-67036710c93f"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:51:11.470219 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.470184    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9374bef7-0f72-4a04-8c2a-67036710c93f-kube-api-access-qmdqp" (OuterVolumeSpecName: "kube-api-access-qmdqp") pod "9374bef7-0f72-4a04-8c2a-67036710c93f" (UID: "9374bef7-0f72-4a04-8c2a-67036710c93f"). InnerVolumeSpecName "kube-api-access-qmdqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:51:11.470341 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.470260    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9374bef7-0f72-4a04-8c2a-67036710c93f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9374bef7-0f72-4a04-8c2a-67036710c93f" (UID: "9374bef7-0f72-4a04-8c2a-67036710c93f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:51:11.481189 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.481161    2549 generic.go:358] "Generic (PLEG): container finished" podID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerID="63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f" exitCode=0
May 25 13:51:11.481325 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.481242    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"
May 25 13:51:11.481325 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.481238    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" event={"ID":"9374bef7-0f72-4a04-8c2a-67036710c93f","Type":"ContainerDied","Data":"63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f"}
May 25 13:51:11.481445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.481324    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg" event={"ID":"9374bef7-0f72-4a04-8c2a-67036710c93f","Type":"ContainerDied","Data":"004445c0b6d4f3a26003bf8a0b4f4b2c6777b0283942ae61592511a3b7b9027f"}
May 25 13:51:11.481445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.481347    2549 scope.go:117] "RemoveContainer" containerID="cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58"
May 25 13:51:11.482728 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.482706    2549 generic.go:358] "Generic (PLEG): container finished" podID="19c42715-506c-4fa3-95e8-89c70c818c94" containerID="1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2" exitCode=0
May 25 13:51:11.482845 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.482740    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" event={"ID":"19c42715-506c-4fa3-95e8-89c70c818c94","Type":"ContainerDied","Data":"1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2"}
May 25 13:51:11.490168 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.490152    2549 scope.go:117] "RemoveContainer" containerID="63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f"
May 25 13:51:11.497967 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.497947    2549 scope.go:117] "RemoveContainer" containerID="d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93"
May 25 13:51:11.507867 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.507839    2549 scope.go:117] "RemoveContainer" containerID="cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58"
May 25 13:51:11.508407 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:51:11.508384    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58\": container with ID starting with cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58 not found: ID does not exist" containerID="cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58"
May 25 13:51:11.508516 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.508419    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58"} err="failed to get container status \"cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58\": rpc error: code = NotFound desc = could not find container \"cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58\": container with ID starting with cf7fc6e465944eb9696bb686c2ea2c80b56d99c469ab369d68e0b15eff75ae58 not found: ID does not exist"
May 25 13:51:11.508516 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.508443    2549 scope.go:117] "RemoveContainer" containerID="63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f"
May 25 13:51:11.508727 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:51:11.508709    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f\": container with ID starting with 63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f not found: ID does not exist" containerID="63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f"
May 25 13:51:11.508809 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.508736    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f"} err="failed to get container status \"63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f\": rpc error: code = NotFound desc = could not find container \"63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f\": container with ID starting with 63d6ac0128a3417305227ac9d2ff4370f1b813ad9701e979af34cf2225af609f not found: ID does not exist"
May 25 13:51:11.508809 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.508758    2549 scope.go:117] "RemoveContainer" containerID="d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93"
May 25 13:51:11.509000 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:51:11.508982    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93\": container with ID starting with d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93 not found: ID does not exist" containerID="d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93"
May 25 13:51:11.509042 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.509005    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93"} err="failed to get container status \"d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93\": rpc error: code = NotFound desc = could not find container \"d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93\": container with ID starting with d0cf68da312509be3f1e5c4d3ef838d58037e023bcd43d536104f6d019d2ee93 not found: ID does not exist"
May 25 13:51:11.518291 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.518248    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"]
May 25 13:51:11.521448 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.521423    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-jwfbg"]
May 25 13:51:11.569142 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.569108    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9374bef7-0f72-4a04-8c2a-67036710c93f-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:51:11.569142 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.569139    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmdqp\" (UniqueName: \"kubernetes.io/projected/9374bef7-0f72-4a04-8c2a-67036710c93f-kube-api-access-qmdqp\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:51:11.569367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.569152    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9374bef7-0f72-4a04-8c2a-67036710c93f-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:51:11.569367 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:11.569168    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9374bef7-0f72-4a04-8c2a-67036710c93f-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:51:12.487615 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:12.487580    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" event={"ID":"19c42715-506c-4fa3-95e8-89c70c818c94","Type":"ContainerStarted","Data":"45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3"}
May 25 13:51:12.488012 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:12.487626    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" event={"ID":"19c42715-506c-4fa3-95e8-89c70c818c94","Type":"ContainerStarted","Data":"b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626"}
May 25 13:51:12.488012 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:12.487873    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:12.488012 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:12.487900    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:12.495390 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:12.495367    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" path="/var/lib/kubelet/pods/9374bef7-0f72-4a04-8c2a-67036710c93f/volumes"
May 25 13:51:12.505362 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:12.505322    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podStartSLOduration=6.505311559 podStartE2EDuration="6.505311559s" podCreationTimestamp="2026-05-25 13:51:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:51:12.504895895 +0000 UTC m=+2822.543653019" watchObservedRunningTime="2026-05-25 13:51:12.505311559 +0000 UTC m=+2822.544068682"
May 25 13:51:18.498656 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:18.498629    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:51:48.499238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:48.499194    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:51:58.499521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:51:58.499479    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:52:08.500184 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:08.500131    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:52:18.500198 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:18.500156    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:52:28.502519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:28.502488    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:52:36.704125 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.704082    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"]
May 25 13:52:36.704576 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.704439    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container" containerID="cri-o://b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626" gracePeriod=30
May 25 13:52:36.704576 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.704475    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kube-rbac-proxy" containerID="cri-o://45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3" gracePeriod=30
May 25 13:52:36.811239 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811199    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"]
May 25 13:52:36.811501 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811487    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kserve-container"
May 25 13:52:36.811501 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811502    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kserve-container"
May 25 13:52:36.811593 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811519    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="storage-initializer"
May 25 13:52:36.811593 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811525    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="storage-initializer"
May 25 13:52:36.811593 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811532    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kube-rbac-proxy"
May 25 13:52:36.811593 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811537    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kube-rbac-proxy"
May 25 13:52:36.811593 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811586    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kserve-container"
May 25 13:52:36.811593 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.811594    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="9374bef7-0f72-4a04-8c2a-67036710c93f" containerName="kube-rbac-proxy"
May 25 13:52:36.815799 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.815774    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:36.818197 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.818167    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
May 25 13:52:36.818358 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.818179    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
May 25 13:52:36.825522 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.825497    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"]
May 25 13:52:36.910968 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.910908    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c572bb06-2f8d-4fde-b095-4592f7956f81-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:36.911224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.911015    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c572bb06-2f8d-4fde-b095-4592f7956f81-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:36.911224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.911053    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzwf\" (UniqueName: \"kubernetes.io/projected/c572bb06-2f8d-4fde-b095-4592f7956f81-kube-api-access-zlzwf\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:36.911224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:36.911134    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c572bb06-2f8d-4fde-b095-4592f7956f81-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.012505 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.012386    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c572bb06-2f8d-4fde-b095-4592f7956f81-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.012505 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.012456    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c572bb06-2f8d-4fde-b095-4592f7956f81-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.012505 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.012489    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzwf\" (UniqueName: \"kubernetes.io/projected/c572bb06-2f8d-4fde-b095-4592f7956f81-kube-api-access-zlzwf\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.012795 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.012533    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c572bb06-2f8d-4fde-b095-4592f7956f81-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.012850 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.012823    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c572bb06-2f8d-4fde-b095-4592f7956f81-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.013193 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.013169    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c572bb06-2f8d-4fde-b095-4592f7956f81-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.015089 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.015070    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c572bb06-2f8d-4fde-b095-4592f7956f81-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.021874 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.021852    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzwf\" (UniqueName: \"kubernetes.io/projected/c572bb06-2f8d-4fde-b095-4592f7956f81-kube-api-access-zlzwf\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.128347 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.128290    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:37.257662 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.257629    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"]
May 25 13:52:37.261109 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:52:37.261075    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc572bb06_2f8d_4fde_b095_4592f7956f81.slice/crio-7fde8044cb8ef6c953aee0cfe325c7459580f81ff1ee513da2b7e932a1698f9d WatchSource:0}: Error finding container 7fde8044cb8ef6c953aee0cfe325c7459580f81ff1ee513da2b7e932a1698f9d: Status 404 returned error can't find the container with id 7fde8044cb8ef6c953aee0cfe325c7459580f81ff1ee513da2b7e932a1698f9d
May 25 13:52:37.262886 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.262845    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:52:37.724923 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.724877    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" event={"ID":"c572bb06-2f8d-4fde-b095-4592f7956f81","Type":"ContainerStarted","Data":"b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369"}
May 25 13:52:37.724923 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.724913    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" event={"ID":"c572bb06-2f8d-4fde-b095-4592f7956f81","Type":"ContainerStarted","Data":"7fde8044cb8ef6c953aee0cfe325c7459580f81ff1ee513da2b7e932a1698f9d"}
May 25 13:52:37.726842 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.726809    2549 generic.go:358] "Generic (PLEG): container finished" podID="19c42715-506c-4fa3-95e8-89c70c818c94" containerID="45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3" exitCode=2
May 25 13:52:37.726966 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:37.726894    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" event={"ID":"19c42715-506c-4fa3-95e8-89c70c818c94","Type":"ContainerDied","Data":"45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3"}
May 25 13:52:38.493093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:38.493047    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
May 25 13:52:38.499900 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:38.499867    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 13:52:41.548449 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.548422    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:52:41.649162 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.649036    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c42715-506c-4fa3-95e8-89c70c818c94-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"19c42715-506c-4fa3-95e8-89c70c818c94\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") "
May 25 13:52:41.649162 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.649141    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c42715-506c-4fa3-95e8-89c70c818c94-kserve-provision-location\") pod \"19c42715-506c-4fa3-95e8-89c70c818c94\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") "
May 25 13:52:41.649457 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.649194    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkcwd\" (UniqueName: \"kubernetes.io/projected/19c42715-506c-4fa3-95e8-89c70c818c94-kube-api-access-vkcwd\") pod \"19c42715-506c-4fa3-95e8-89c70c818c94\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") "
May 25 13:52:41.649457 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.649224    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c42715-506c-4fa3-95e8-89c70c818c94-proxy-tls\") pod \"19c42715-506c-4fa3-95e8-89c70c818c94\" (UID: \"19c42715-506c-4fa3-95e8-89c70c818c94\") "
May 25 13:52:41.649572 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.649512    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c42715-506c-4fa3-95e8-89c70c818c94-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "19c42715-506c-4fa3-95e8-89c70c818c94" (UID: "19c42715-506c-4fa3-95e8-89c70c818c94"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:52:41.649640 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.649595    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c42715-506c-4fa3-95e8-89c70c818c94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19c42715-506c-4fa3-95e8-89c70c818c94" (UID: "19c42715-506c-4fa3-95e8-89c70c818c94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:52:41.651456 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.651418    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c42715-506c-4fa3-95e8-89c70c818c94-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19c42715-506c-4fa3-95e8-89c70c818c94" (UID: "19c42715-506c-4fa3-95e8-89c70c818c94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:52:41.651569 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.651506    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c42715-506c-4fa3-95e8-89c70c818c94-kube-api-access-vkcwd" (OuterVolumeSpecName: "kube-api-access-vkcwd") pod "19c42715-506c-4fa3-95e8-89c70c818c94" (UID: "19c42715-506c-4fa3-95e8-89c70c818c94"). InnerVolumeSpecName "kube-api-access-vkcwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:52:41.741062 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.741015    2549 generic.go:358] "Generic (PLEG): container finished" podID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerID="b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369" exitCode=0
May 25 13:52:41.741245 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.741088    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" event={"ID":"c572bb06-2f8d-4fde-b095-4592f7956f81","Type":"ContainerDied","Data":"b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369"}
May 25 13:52:41.742951 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.742926    2549 generic.go:358] "Generic (PLEG): container finished" podID="19c42715-506c-4fa3-95e8-89c70c818c94" containerID="b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626" exitCode=0
May 25 13:52:41.743045 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.742992    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"
May 25 13:52:41.743084 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.742993    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" event={"ID":"19c42715-506c-4fa3-95e8-89c70c818c94","Type":"ContainerDied","Data":"b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626"}
May 25 13:52:41.743119 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.743095    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j" event={"ID":"19c42715-506c-4fa3-95e8-89c70c818c94","Type":"ContainerDied","Data":"9421300bab8a760948e3201ed35f3e4cd2b98e3f7bfde193bf0931c2c875ed26"}
May 25 13:52:41.743119 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.743111    2549 scope.go:117] "RemoveContainer" containerID="45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3"
May 25 13:52:41.750341 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.750322    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkcwd\" (UniqueName: \"kubernetes.io/projected/19c42715-506c-4fa3-95e8-89c70c818c94-kube-api-access-vkcwd\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:52:41.750440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.750347    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c42715-506c-4fa3-95e8-89c70c818c94-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:52:41.750440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.750365    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c42715-506c-4fa3-95e8-89c70c818c94-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:52:41.750440 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.750404    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c42715-506c-4fa3-95e8-89c70c818c94-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:52:41.751497 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.751442    2549 scope.go:117] "RemoveContainer" containerID="b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626"
May 25 13:52:41.758482 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.758461    2549 scope.go:117] "RemoveContainer" containerID="1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2"
May 25 13:52:41.766503 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.766483    2549 scope.go:117] "RemoveContainer" containerID="45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3"
May 25 13:52:41.766781 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:52:41.766764    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3\": container with ID starting with 45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3 not found: ID does not exist" containerID="45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3"
May 25 13:52:41.766835 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.766790    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3"} err="failed to get container status \"45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3\": rpc error: code = NotFound desc = could not find container \"45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3\": container with ID starting with 45104fe85aba65552d1dc24d3ab5c01a984f1557cfbca51ebed922955bf8e4e3 not found: ID does not exist"
May 25 13:52:41.766835 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.766808    2549 scope.go:117] "RemoveContainer" containerID="b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626"
May 25 13:52:41.767051 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:52:41.767035    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626\": container with ID starting with b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626 not found: ID does not exist" containerID="b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626"
May 25 13:52:41.767102 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.767057    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626"} err="failed to get container status \"b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626\": rpc error: code = NotFound desc = could not find container \"b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626\": container with ID starting with b1863951fe90aa9c8f79317692a6f99baed8807a3934f7a1a3bc45223d6e1626 not found: ID does not exist"
May 25 13:52:41.767102 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.767072    2549 scope.go:117] "RemoveContainer" containerID="1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2"
May 25 13:52:41.767335 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:52:41.767312    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2\": container with ID starting with 1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2 not found: ID does not exist" containerID="1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2"
May 25 13:52:41.767435 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.767337    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2"} err="failed to get container status \"1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2\": rpc error: code = NotFound desc = could not find container \"1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2\": container with ID starting with 1eccf0abe4f1d22dc32030be35d1fd5f38f3c1541059fe051810fbd0bc0b98b2 not found: ID does not exist"
May 25 13:52:41.771899 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.771875    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"]
May 25 13:52:41.775839 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:41.775818    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-g648j"]
May 25 13:52:42.495757 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:42.495721    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" path="/var/lib/kubelet/pods/19c42715-506c-4fa3-95e8-89c70c818c94/volumes"
May 25 13:52:42.748206 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:42.748117    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" event={"ID":"c572bb06-2f8d-4fde-b095-4592f7956f81","Type":"ContainerStarted","Data":"6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab"}
May 25 13:52:42.748206 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:42.748158    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" event={"ID":"c572bb06-2f8d-4fde-b095-4592f7956f81","Type":"ContainerStarted","Data":"de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1"}
May 25 13:52:42.748635 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:42.748360    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:42.767051 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:42.766996    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podStartSLOduration=6.766979329 podStartE2EDuration="6.766979329s" podCreationTimestamp="2026-05-25 13:52:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:52:42.766832522 +0000 UTC m=+2912.805589646" watchObservedRunningTime="2026-05-25 13:52:42.766979329 +0000 UTC m=+2912.805736455"
May 25 13:52:43.751163 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:43.751130    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:52:49.758705 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:52:49.758676    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:53:19.759836 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:53:19.759742    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:53:29.759853 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:53:29.759804    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:53:39.759577 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:53:39.759536    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:53:49.759503 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:53:49.759457    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:53:59.763224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:53:59.763182    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:54:06.920692 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:06.920655    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"]
May 25 13:54:06.921224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:06.920994    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container" containerID="cri-o://de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1" gracePeriod=30
May 25 13:54:06.921224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:06.921053    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kube-rbac-proxy" containerID="cri-o://6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab" gracePeriod=30
May 25 13:54:07.984116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:07.984081    2549 generic.go:358] "Generic (PLEG): container finished" podID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerID="6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab" exitCode=2
May 25 13:54:07.984530 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:07.984137    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" event={"ID":"c572bb06-2f8d-4fde-b095-4592f7956f81","Type":"ContainerDied","Data":"6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab"}
May 25 13:54:09.138380 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138342    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"]
May 25 13:54:09.138759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138605    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container"
May 25 13:54:09.138759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138615    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container"
May 25 13:54:09.138759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138633    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="storage-initializer"
May 25 13:54:09.138759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138640    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="storage-initializer"
May 25 13:54:09.138759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138648    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kube-rbac-proxy"
May 25 13:54:09.138759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138654    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kube-rbac-proxy"
May 25 13:54:09.138759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138694    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kube-rbac-proxy"
May 25 13:54:09.138759 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.138701    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="19c42715-506c-4fa3-95e8-89c70c818c94" containerName="kserve-container"
May 25 13:54:09.141640 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.141622    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.143865 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.143843    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
May 25 13:54:09.144123 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.144109    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
May 25 13:54:09.152445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.152423    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"]
May 25 13:54:09.214373 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.214335    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhgst\" (UniqueName: \"kubernetes.io/projected/24fac400-5d47-4beb-a8c4-060017b7a63b-kube-api-access-rhgst\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.214373 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.214375    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fac400-5d47-4beb-a8c4-060017b7a63b-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.214580 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.214411    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.214580 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.214453    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fac400-5d47-4beb-a8c4-060017b7a63b-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.315093 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.315050    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fac400-5d47-4beb-a8c4-060017b7a63b-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.315265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.315114    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rhgst\" (UniqueName: \"kubernetes.io/projected/24fac400-5d47-4beb-a8c4-060017b7a63b-kube-api-access-rhgst\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.315265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.315148    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fac400-5d47-4beb-a8c4-060017b7a63b-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.315265 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.315178    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.315485 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:54:09.315310    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
May 25 13:54:09.315485 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:54:09.315387    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls podName:24fac400-5d47-4beb-a8c4-060017b7a63b nodeName:}" failed. No retries permitted until 2026-05-25 13:54:09.815367203 +0000 UTC m=+2999.854124309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls") pod "isvc-sklearn-predictor-bc9cf9b57-pxqqn" (UID: "24fac400-5d47-4beb-a8c4-060017b7a63b") : secret "isvc-sklearn-predictor-serving-cert" not found
May 25 13:54:09.315643 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.315621    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fac400-5d47-4beb-a8c4-060017b7a63b-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.315748 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.315729    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fac400-5d47-4beb-a8c4-060017b7a63b-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.324432 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.324412    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhgst\" (UniqueName: \"kubernetes.io/projected/24fac400-5d47-4beb-a8c4-060017b7a63b-kube-api-access-rhgst\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.754445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.754402    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 25 13:54:09.759382 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.759350    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 13:54:09.819665 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.819628    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:09.822114 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:09.822084    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-pxqqn\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:10.051342 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:10.051220    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:10.179833 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:10.179769    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"]
May 25 13:54:10.182342 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:54:10.182311    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fac400_5d47_4beb_a8c4_060017b7a63b.slice/crio-8ea92474a7986e34803578a281b3c300c36ac5b7d7acc08a62b49363e38e9d0b WatchSource:0}: Error finding container 8ea92474a7986e34803578a281b3c300c36ac5b7d7acc08a62b49363e38e9d0b: Status 404 returned error can't find the container with id 8ea92474a7986e34803578a281b3c300c36ac5b7d7acc08a62b49363e38e9d0b
May 25 13:54:10.993933 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:10.993895    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" event={"ID":"24fac400-5d47-4beb-a8c4-060017b7a63b","Type":"ContainerStarted","Data":"b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc"}
May 25 13:54:10.993933 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:10.993932    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" event={"ID":"24fac400-5d47-4beb-a8c4-060017b7a63b","Type":"ContainerStarted","Data":"8ea92474a7986e34803578a281b3c300c36ac5b7d7acc08a62b49363e38e9d0b"}
May 25 13:54:11.176963 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:11.176937    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:54:11.181844 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:11.181821    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:54:12.867334 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:12.867309    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:54:13.000952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.000858    2549 generic.go:358] "Generic (PLEG): container finished" podID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerID="de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1" exitCode=0
May 25 13:54:13.000952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.000906    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" event={"ID":"c572bb06-2f8d-4fde-b095-4592f7956f81","Type":"ContainerDied","Data":"de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1"}
May 25 13:54:13.000952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.000936    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk" event={"ID":"c572bb06-2f8d-4fde-b095-4592f7956f81","Type":"ContainerDied","Data":"7fde8044cb8ef6c953aee0cfe325c7459580f81ff1ee513da2b7e932a1698f9d"}
May 25 13:54:13.000952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.000936    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"
May 25 13:54:13.000952 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.000953    2549 scope.go:117] "RemoveContainer" containerID="6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab"
May 25 13:54:13.008810 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.008794    2549 scope.go:117] "RemoveContainer" containerID="de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1"
May 25 13:54:13.015621 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.015603    2549 scope.go:117] "RemoveContainer" containerID="b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369"
May 25 13:54:13.022185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.022167    2549 scope.go:117] "RemoveContainer" containerID="6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab"
May 25 13:54:13.022449 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:54:13.022432    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab\": container with ID starting with 6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab not found: ID does not exist" containerID="6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab"
May 25 13:54:13.022533 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.022458    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab"} err="failed to get container status \"6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab\": rpc error: code = NotFound desc = could not find container \"6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab\": container with ID starting with 6ee6d43d908be72a716312315ec75264b5f5dcb8c8f2cb66f5b4e5cd38fe1aab not found: ID does not exist"
May 25 13:54:13.022533 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.022475    2549 scope.go:117] "RemoveContainer" containerID="de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1"
May 25 13:54:13.022707 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:54:13.022689    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1\": container with ID starting with de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1 not found: ID does not exist" containerID="de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1"
May 25 13:54:13.022747 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.022717    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1"} err="failed to get container status \"de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1\": rpc error: code = NotFound desc = could not find container \"de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1\": container with ID starting with de791d3c83e68bc43a3d1e593d093ecb3e37381cd0df0526bcfa884823e8f2a1 not found: ID does not exist"
May 25 13:54:13.022747 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.022738    2549 scope.go:117] "RemoveContainer" containerID="b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369"
May 25 13:54:13.022937 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:54:13.022922    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369\": container with ID starting with b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369 not found: ID does not exist" containerID="b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369"
May 25 13:54:13.022978 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.022942    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369"} err="failed to get container status \"b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369\": rpc error: code = NotFound desc = could not find container \"b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369\": container with ID starting with b52d7204f375db46009a5d1e0015375b56853592115de17edd221e13719c2369 not found: ID does not exist"
May 25 13:54:13.048254 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.048216    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c572bb06-2f8d-4fde-b095-4592f7956f81-proxy-tls\") pod \"c572bb06-2f8d-4fde-b095-4592f7956f81\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") "
May 25 13:54:13.048405 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.048310    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlzwf\" (UniqueName: \"kubernetes.io/projected/c572bb06-2f8d-4fde-b095-4592f7956f81-kube-api-access-zlzwf\") pod \"c572bb06-2f8d-4fde-b095-4592f7956f81\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") "
May 25 13:54:13.048405 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.048340    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c572bb06-2f8d-4fde-b095-4592f7956f81-kserve-provision-location\") pod \"c572bb06-2f8d-4fde-b095-4592f7956f81\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") "
May 25 13:54:13.048405 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.048363    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c572bb06-2f8d-4fde-b095-4592f7956f81-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"c572bb06-2f8d-4fde-b095-4592f7956f81\" (UID: \"c572bb06-2f8d-4fde-b095-4592f7956f81\") "
May 25 13:54:13.048746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.048718    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c572bb06-2f8d-4fde-b095-4592f7956f81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c572bb06-2f8d-4fde-b095-4592f7956f81" (UID: "c572bb06-2f8d-4fde-b095-4592f7956f81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:54:13.048819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.048751    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c572bb06-2f8d-4fde-b095-4592f7956f81-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "c572bb06-2f8d-4fde-b095-4592f7956f81" (UID: "c572bb06-2f8d-4fde-b095-4592f7956f81"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:54:13.050512 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.050482    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c572bb06-2f8d-4fde-b095-4592f7956f81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c572bb06-2f8d-4fde-b095-4592f7956f81" (UID: "c572bb06-2f8d-4fde-b095-4592f7956f81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:54:13.050512 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.050485    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c572bb06-2f8d-4fde-b095-4592f7956f81-kube-api-access-zlzwf" (OuterVolumeSpecName: "kube-api-access-zlzwf") pod "c572bb06-2f8d-4fde-b095-4592f7956f81" (UID: "c572bb06-2f8d-4fde-b095-4592f7956f81"). InnerVolumeSpecName "kube-api-access-zlzwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:54:13.149098 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.149057    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c572bb06-2f8d-4fde-b095-4592f7956f81-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:54:13.149098 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.149092    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c572bb06-2f8d-4fde-b095-4592f7956f81-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:54:13.149098 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.149104    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c572bb06-2f8d-4fde-b095-4592f7956f81-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:54:13.149382 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.149117    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlzwf\" (UniqueName: \"kubernetes.io/projected/c572bb06-2f8d-4fde-b095-4592f7956f81-kube-api-access-zlzwf\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:54:13.323238 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.323196    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"]
May 25 13:54:13.328939 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:13.328913    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-944jk"]
May 25 13:54:14.005147 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:14.005117    2549 generic.go:358] "Generic (PLEG): container finished" podID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerID="b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc" exitCode=0
May 25 13:54:14.005584 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:14.005192    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" event={"ID":"24fac400-5d47-4beb-a8c4-060017b7a63b","Type":"ContainerDied","Data":"b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc"}
May 25 13:54:14.497444 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:14.497408    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" path="/var/lib/kubelet/pods/c572bb06-2f8d-4fde-b095-4592f7956f81/volumes"
May 25 13:54:15.010828 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:15.010795    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" event={"ID":"24fac400-5d47-4beb-a8c4-060017b7a63b","Type":"ContainerStarted","Data":"0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983"}
May 25 13:54:15.011219 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:15.010836    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" event={"ID":"24fac400-5d47-4beb-a8c4-060017b7a63b","Type":"ContainerStarted","Data":"80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e"}
May 25 13:54:15.011219 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:15.011116    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:15.011328 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:15.011229    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:15.012523 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:15.012497    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:54:15.032787 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:15.032742    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podStartSLOduration=6.032727746 podStartE2EDuration="6.032727746s" podCreationTimestamp="2026-05-25 13:54:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:54:15.029949306 +0000 UTC m=+3005.068706432" watchObservedRunningTime="2026-05-25 13:54:15.032727746 +0000 UTC m=+3005.071484930"
May 25 13:54:16.013811 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:16.013764    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:54:21.018028 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:21.017999    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:54:21.018624 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:21.018599    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:54:31.018925 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:31.018883    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:54:41.018767 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:41.018728    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:54:51.019589 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:54:51.019503    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:55:01.018938 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:01.018891    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:55:11.018610 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:11.018515    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:55:21.019457 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:21.019429    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:55:29.232703 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.232659    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"]
May 25 13:55:29.233252 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.232992    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" containerID="cri-o://80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e" gracePeriod=30
May 25 13:55:29.233252 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.233110    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kube-rbac-proxy" containerID="cri-o://0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983" gracePeriod=30
May 25 13:55:29.385244 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385207    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"]
May 25 13:55:29.385531 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385515    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kube-rbac-proxy"
May 25 13:55:29.385531 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385532    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kube-rbac-proxy"
May 25 13:55:29.385617 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385542    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="storage-initializer"
May 25 13:55:29.385617 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385548    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="storage-initializer"
May 25 13:55:29.385617 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385559    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container"
May 25 13:55:29.385617 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385566    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container"
May 25 13:55:29.385763 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385630    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kserve-container"
May 25 13:55:29.385763 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.385638    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="c572bb06-2f8d-4fde-b095-4592f7956f81" containerName="kube-rbac-proxy"
May 25 13:55:29.388532 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.388517    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.392107 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.392086    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 13:55:29.392218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.392089    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
May 25 13:55:29.409286 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.409242    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"]
May 25 13:55:29.498481 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.498378    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.498481 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.498417    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d76a4f-4432-4607-a977-6b6bfc7e704e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.498481 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.498446    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjscj\" (UniqueName: \"kubernetes.io/projected/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kube-api-access-hjscj\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.498770 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.498560    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d76a4f-4432-4607-a977-6b6bfc7e704e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.599828 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.599781    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.599828 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.599823    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d76a4f-4432-4607-a977-6b6bfc7e704e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.600057 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.599844    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hjscj\" (UniqueName: \"kubernetes.io/projected/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kube-api-access-hjscj\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.600057 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.599899    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d76a4f-4432-4607-a977-6b6bfc7e704e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.600259 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.600232    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.600558 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.600539    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d76a4f-4432-4607-a977-6b6bfc7e704e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.602470 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.602450    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d76a4f-4432-4607-a977-6b6bfc7e704e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.608640 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.608617    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjscj\" (UniqueName: \"kubernetes.io/projected/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kube-api-access-hjscj\") pod \"sklearn-v2-mlserver-predictor-65d8664766-ldlf7\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.698687 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.698641    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:29.824116 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:29.824080    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"]
May 25 13:55:29.827218 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:55:29.827192    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d76a4f_4432_4607_a977_6b6bfc7e704e.slice/crio-b5b0b8815d2bf6b44c435cb87f251209a8215b025dd705871ac32cd59eaaa294 WatchSource:0}: Error finding container b5b0b8815d2bf6b44c435cb87f251209a8215b025dd705871ac32cd59eaaa294: Status 404 returned error can't find the container with id b5b0b8815d2bf6b44c435cb87f251209a8215b025dd705871ac32cd59eaaa294
May 25 13:55:30.217568 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:30.217529    2549 generic.go:358] "Generic (PLEG): container finished" podID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerID="0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983" exitCode=2
May 25 13:55:30.217773 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:30.217608    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" event={"ID":"24fac400-5d47-4beb-a8c4-060017b7a63b","Type":"ContainerDied","Data":"0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983"}
May 25 13:55:30.219020 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:30.218991    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" event={"ID":"a2d76a4f-4432-4607-a977-6b6bfc7e704e","Type":"ContainerStarted","Data":"555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0"}
May 25 13:55:30.219147 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:30.219027    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" event={"ID":"a2d76a4f-4432-4607-a977-6b6bfc7e704e","Type":"ContainerStarted","Data":"b5b0b8815d2bf6b44c435cb87f251209a8215b025dd705871ac32cd59eaaa294"}
May 25 13:55:31.014587 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:31.014543    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
May 25 13:55:31.018777 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:31.018752    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 13:55:33.673210 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.673186    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:55:33.729138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.729049    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fac400-5d47-4beb-a8c4-060017b7a63b-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"24fac400-5d47-4beb-a8c4-060017b7a63b\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") "
May 25 13:55:33.729138 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.729131    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fac400-5d47-4beb-a8c4-060017b7a63b-kserve-provision-location\") pod \"24fac400-5d47-4beb-a8c4-060017b7a63b\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") "
May 25 13:55:33.729393 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.729165    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhgst\" (UniqueName: \"kubernetes.io/projected/24fac400-5d47-4beb-a8c4-060017b7a63b-kube-api-access-rhgst\") pod \"24fac400-5d47-4beb-a8c4-060017b7a63b\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") "
May 25 13:55:33.729393 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.729194    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls\") pod \"24fac400-5d47-4beb-a8c4-060017b7a63b\" (UID: \"24fac400-5d47-4beb-a8c4-060017b7a63b\") "
May 25 13:55:33.729510 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.729478    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fac400-5d47-4beb-a8c4-060017b7a63b-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "24fac400-5d47-4beb-a8c4-060017b7a63b" (UID: "24fac400-5d47-4beb-a8c4-060017b7a63b"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:55:33.729569 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.729501    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fac400-5d47-4beb-a8c4-060017b7a63b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24fac400-5d47-4beb-a8c4-060017b7a63b" (UID: "24fac400-5d47-4beb-a8c4-060017b7a63b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:55:33.731514 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.731488    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24fac400-5d47-4beb-a8c4-060017b7a63b" (UID: "24fac400-5d47-4beb-a8c4-060017b7a63b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:55:33.731514 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.731506    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fac400-5d47-4beb-a8c4-060017b7a63b-kube-api-access-rhgst" (OuterVolumeSpecName: "kube-api-access-rhgst") pod "24fac400-5d47-4beb-a8c4-060017b7a63b" (UID: "24fac400-5d47-4beb-a8c4-060017b7a63b"). InnerVolumeSpecName "kube-api-access-rhgst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:55:33.830209 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.830156    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhgst\" (UniqueName: \"kubernetes.io/projected/24fac400-5d47-4beb-a8c4-060017b7a63b-kube-api-access-rhgst\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:55:33.830209 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.830204    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24fac400-5d47-4beb-a8c4-060017b7a63b-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:55:33.830209 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.830219    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24fac400-5d47-4beb-a8c4-060017b7a63b-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:55:33.830209 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:33.830228    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24fac400-5d47-4beb-a8c4-060017b7a63b-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:55:34.231425 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.231383    2549 generic.go:358] "Generic (PLEG): container finished" podID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerID="80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e" exitCode=0
May 25 13:55:34.231592 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.231472    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" event={"ID":"24fac400-5d47-4beb-a8c4-060017b7a63b","Type":"ContainerDied","Data":"80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e"}
May 25 13:55:34.231592 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.231504    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn" event={"ID":"24fac400-5d47-4beb-a8c4-060017b7a63b","Type":"ContainerDied","Data":"8ea92474a7986e34803578a281b3c300c36ac5b7d7acc08a62b49363e38e9d0b"}
May 25 13:55:34.231592 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.231524    2549 scope.go:117] "RemoveContainer" containerID="0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983"
May 25 13:55:34.231592 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.231536    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"
May 25 13:55:34.233087 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.233059    2549 generic.go:358] "Generic (PLEG): container finished" podID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerID="555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0" exitCode=0
May 25 13:55:34.233219 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.233138    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" event={"ID":"a2d76a4f-4432-4607-a977-6b6bfc7e704e","Type":"ContainerDied","Data":"555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0"}
May 25 13:55:34.240147 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.240024    2549 scope.go:117] "RemoveContainer" containerID="80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e"
May 25 13:55:34.247106 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.247078    2549 scope.go:117] "RemoveContainer" containerID="b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc"
May 25 13:55:34.255488 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.255468    2549 scope.go:117] "RemoveContainer" containerID="0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983"
May 25 13:55:34.255776 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:55:34.255755    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983\": container with ID starting with 0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983 not found: ID does not exist" containerID="0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983"
May 25 13:55:34.255866 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.255782    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983"} err="failed to get container status \"0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983\": rpc error: code = NotFound desc = could not find container \"0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983\": container with ID starting with 0b171f4c7c50e36e38ee3d937032e065bf2f73bf702d85ff1c1b2daf44d90983 not found: ID does not exist"
May 25 13:55:34.255866 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.255802    2549 scope.go:117] "RemoveContainer" containerID="80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e"
May 25 13:55:34.256079 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:55:34.256061    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e\": container with ID starting with 80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e not found: ID does not exist" containerID="80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e"
May 25 13:55:34.256122 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.256086    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e"} err="failed to get container status \"80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e\": rpc error: code = NotFound desc = could not find container \"80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e\": container with ID starting with 80e154b2b666d3de97d8aa32936ca274bfc52da608cc906498b0341f9373e05e not found: ID does not exist"
May 25 13:55:34.256122 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.256102    2549 scope.go:117] "RemoveContainer" containerID="b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc"
May 25 13:55:34.256410 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:55:34.256385    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc\": container with ID starting with b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc not found: ID does not exist" containerID="b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc"
May 25 13:55:34.256494 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.256416    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc"} err="failed to get container status \"b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc\": rpc error: code = NotFound desc = could not find container \"b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc\": container with ID starting with b2d15313f9024c74055919d0440b0b528f14fdaf2d6dcdf9ee2d29ad8de38ddc not found: ID does not exist"
May 25 13:55:34.266030 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.266004    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"]
May 25 13:55:34.269990 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.269784    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pxqqn"]
May 25 13:55:34.496289 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:34.496193    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" path="/var/lib/kubelet/pods/24fac400-5d47-4beb-a8c4-060017b7a63b/volumes"
May 25 13:55:35.238965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:35.238934    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" event={"ID":"a2d76a4f-4432-4607-a977-6b6bfc7e704e","Type":"ContainerStarted","Data":"67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2"}
May 25 13:55:35.238965 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:35.238970    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" event={"ID":"a2d76a4f-4432-4607-a977-6b6bfc7e704e","Type":"ContainerStarted","Data":"548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe"}
May 25 13:55:35.239415 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:35.239162    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:35.258615 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:35.258561    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" podStartSLOduration=6.258542971 podStartE2EDuration="6.258542971s" podCreationTimestamp="2026-05-25 13:55:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:55:35.25672589 +0000 UTC m=+3085.295483047" watchObservedRunningTime="2026-05-25 13:55:35.258542971 +0000 UTC m=+3085.297300128"
May 25 13:55:36.244308 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:36.243877    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:55:42.251590 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:55:42.251558    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:56:12.292318 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:12.292244    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 13:56:22.255033 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:22.254944    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:56:29.443617 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.443580    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"]
May 25 13:56:29.444188 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.444000    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kserve-container" containerID="cri-o://548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe" gracePeriod=30
May 25 13:56:29.444188 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.444101    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kube-rbac-proxy" containerID="cri-o://67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2" gracePeriod=30
May 25 13:56:29.537935 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.537897    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"]
May 25 13:56:29.538178 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.538166    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kube-rbac-proxy"
May 25 13:56:29.538222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.538179    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kube-rbac-proxy"
May 25 13:56:29.538222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.538190    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container"
May 25 13:56:29.538222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.538198    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container"
May 25 13:56:29.538222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.538207    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="storage-initializer"
May 25 13:56:29.538222 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.538212    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="storage-initializer"
May 25 13:56:29.538402 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.538253    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kube-rbac-proxy"
May 25 13:56:29.538402 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.538265    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="24fac400-5d47-4beb-a8c4-060017b7a63b" containerName="kserve-container"
May 25 13:56:29.542370 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.542351    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.544671 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.544650    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
May 25 13:56:29.544758 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.544689    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
May 25 13:56:29.551389 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.551361    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"]
May 25 13:56:29.636436 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.636401    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d73991-307c-44a3-93ac-3dc15d8f86ea-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.636631 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.636453    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.636631 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.636484    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6zn\" (UniqueName: \"kubernetes.io/projected/53d73991-307c-44a3-93ac-3dc15d8f86ea-kube-api-access-dv6zn\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.636631 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.636553    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53d73991-307c-44a3-93ac-3dc15d8f86ea-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.737218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.737122    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d73991-307c-44a3-93ac-3dc15d8f86ea-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.737218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.737179    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.737218 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.737211    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6zn\" (UniqueName: \"kubernetes.io/projected/53d73991-307c-44a3-93ac-3dc15d8f86ea-kube-api-access-dv6zn\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.737524 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.737247    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53d73991-307c-44a3-93ac-3dc15d8f86ea-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.737524 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:56:29.737396    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
May 25 13:56:29.737524 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:56:29.737475    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls podName:53d73991-307c-44a3-93ac-3dc15d8f86ea nodeName:}" failed. No retries permitted until 2026-05-25 13:56:30.237453496 +0000 UTC m=+3140.276210598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls") pod "isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" (UID: "53d73991-307c-44a3-93ac-3dc15d8f86ea") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
May 25 13:56:29.737700 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.737679    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53d73991-307c-44a3-93ac-3dc15d8f86ea-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.737907 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.737891    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d73991-307c-44a3-93ac-3dc15d8f86ea-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:29.746384 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:29.746352    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6zn\" (UniqueName: \"kubernetes.io/projected/53d73991-307c-44a3-93ac-3dc15d8f86ea-kube-api-access-dv6zn\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:30.240243 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:30.240194    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:30.242869 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:30.242843    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:30.397815 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:30.397777    2549 generic.go:358] "Generic (PLEG): container finished" podID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerID="67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2" exitCode=2
May 25 13:56:30.397987 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:30.397851    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" event={"ID":"a2d76a4f-4432-4607-a977-6b6bfc7e704e","Type":"ContainerDied","Data":"67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2"}
May 25 13:56:30.454158 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:30.454117    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:30.584104 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:30.584079    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"]
May 25 13:56:30.586871 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:56:30.586840    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d73991_307c_44a3_93ac_3dc15d8f86ea.slice/crio-fd2795cd4384b6b67641d2bafbbe7c06da71dc581d600a09a5051bf095b11d2d WatchSource:0}: Error finding container fd2795cd4384b6b67641d2bafbbe7c06da71dc581d600a09a5051bf095b11d2d: Status 404 returned error can't find the container with id fd2795cd4384b6b67641d2bafbbe7c06da71dc581d600a09a5051bf095b11d2d
May 25 13:56:31.401894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:31.401856    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" event={"ID":"53d73991-307c-44a3-93ac-3dc15d8f86ea","Type":"ContainerStarted","Data":"7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006"}
May 25 13:56:31.401894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:31.401894    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" event={"ID":"53d73991-307c-44a3-93ac-3dc15d8f86ea","Type":"ContainerStarted","Data":"fd2795cd4384b6b67641d2bafbbe7c06da71dc581d600a09a5051bf095b11d2d"}
May 25 13:56:32.246342 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:32.246296    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
May 25 13:56:36.418649 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:36.418561    2549 generic.go:358] "Generic (PLEG): container finished" podID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerID="7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006" exitCode=0
May 25 13:56:36.418649 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:36.418602    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" event={"ID":"53d73991-307c-44a3-93ac-3dc15d8f86ea","Type":"ContainerDied","Data":"7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006"}
May 25 13:56:37.297426 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.297400    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:56:37.398427 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.398391    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d76a4f-4432-4607-a977-6b6bfc7e704e-proxy-tls\") pod \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") "
May 25 13:56:37.398427 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.398431    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d76a4f-4432-4607-a977-6b6bfc7e704e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") "
May 25 13:56:37.398693 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.398459    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kserve-provision-location\") pod \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") "
May 25 13:56:37.398693 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.398500    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjscj\" (UniqueName: \"kubernetes.io/projected/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kube-api-access-hjscj\") pod \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\" (UID: \"a2d76a4f-4432-4607-a977-6b6bfc7e704e\") "
May 25 13:56:37.398812 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.398770    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d76a4f-4432-4607-a977-6b6bfc7e704e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "a2d76a4f-4432-4607-a977-6b6bfc7e704e" (UID: "a2d76a4f-4432-4607-a977-6b6bfc7e704e"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:56:37.398878 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.398836    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2d76a4f-4432-4607-a977-6b6bfc7e704e" (UID: "a2d76a4f-4432-4607-a977-6b6bfc7e704e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:56:37.400626 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.400594    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d76a4f-4432-4607-a977-6b6bfc7e704e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2d76a4f-4432-4607-a977-6b6bfc7e704e" (UID: "a2d76a4f-4432-4607-a977-6b6bfc7e704e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:56:37.400783 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.400759    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kube-api-access-hjscj" (OuterVolumeSpecName: "kube-api-access-hjscj") pod "a2d76a4f-4432-4607-a977-6b6bfc7e704e" (UID: "a2d76a4f-4432-4607-a977-6b6bfc7e704e"). InnerVolumeSpecName "kube-api-access-hjscj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:56:37.423246 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.423204    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" event={"ID":"53d73991-307c-44a3-93ac-3dc15d8f86ea","Type":"ContainerStarted","Data":"255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f"}
May 25 13:56:37.423246 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.423253    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" event={"ID":"53d73991-307c-44a3-93ac-3dc15d8f86ea","Type":"ContainerStarted","Data":"93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45"}
May 25 13:56:37.423815 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.423490    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:37.424937 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.424908    2549 generic.go:358] "Generic (PLEG): container finished" podID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerID="548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe" exitCode=0
May 25 13:56:37.425066 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.424968    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" event={"ID":"a2d76a4f-4432-4607-a977-6b6bfc7e704e","Type":"ContainerDied","Data":"548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe"}
May 25 13:56:37.425066 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.424974    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"
May 25 13:56:37.425066 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.424999    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" event={"ID":"a2d76a4f-4432-4607-a977-6b6bfc7e704e","Type":"ContainerDied","Data":"b5b0b8815d2bf6b44c435cb87f251209a8215b025dd705871ac32cd59eaaa294"}
May 25 13:56:37.425066 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.425021    2549 scope.go:117] "RemoveContainer" containerID="67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2"
May 25 13:56:37.433203 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.433063    2549 scope.go:117] "RemoveContainer" containerID="548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe"
May 25 13:56:37.442264 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.442225    2549 scope.go:117] "RemoveContainer" containerID="555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0"
May 25 13:56:37.443349 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.443258    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" podStartSLOduration=8.44324312 podStartE2EDuration="8.44324312s" podCreationTimestamp="2026-05-25 13:56:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:56:37.44192784 +0000 UTC m=+3147.480684976" watchObservedRunningTime="2026-05-25 13:56:37.44324312 +0000 UTC m=+3147.482000244"
May 25 13:56:37.449834 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.449816    2549 scope.go:117] "RemoveContainer" containerID="67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2"
May 25 13:56:37.450102 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:56:37.450083    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2\": container with ID starting with 67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2 not found: ID does not exist" containerID="67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2"
May 25 13:56:37.450187 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.450115    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2"} err="failed to get container status \"67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2\": rpc error: code = NotFound desc = could not find container \"67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2\": container with ID starting with 67fcd681192d19952bb15e94ea0ff0b33a55402b8cd32f7bdb9488c45921daf2 not found: ID does not exist"
May 25 13:56:37.450187 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.450139    2549 scope.go:117] "RemoveContainer" containerID="548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe"
May 25 13:56:37.450414 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:56:37.450393    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe\": container with ID starting with 548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe not found: ID does not exist" containerID="548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe"
May 25 13:56:37.450485 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.450423    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe"} err="failed to get container status \"548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe\": rpc error: code = NotFound desc = could not find container \"548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe\": container with ID starting with 548487de3ecb1862090783faff867016f28aeb6bd8e4e8acfde8c6d25281edbe not found: ID does not exist"
May 25 13:56:37.450485 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.450445    2549 scope.go:117] "RemoveContainer" containerID="555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0"
May 25 13:56:37.450699 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:56:37.450672    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0\": container with ID starting with 555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0 not found: ID does not exist" containerID="555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0"
May 25 13:56:37.450792 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.450704    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0"} err="failed to get container status \"555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0\": rpc error: code = NotFound desc = could not find container \"555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0\": container with ID starting with 555b185c5ca566eda9ec29dda0a93171b30d212858cc3ccd23ebcaefe47a4fe0 not found: ID does not exist"
May 25 13:56:37.455732 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.455704    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"]
May 25 13:56:37.458303 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.458256    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7"]
May 25 13:56:37.499930 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.499889    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2d76a4f-4432-4607-a977-6b6bfc7e704e-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:56:37.499930 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.499921    2549 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2d76a4f-4432-4607-a977-6b6bfc7e704e-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:56:37.499930 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.499939    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:56:37.500230 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:37.499953    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjscj\" (UniqueName: \"kubernetes.io/projected/a2d76a4f-4432-4607-a977-6b6bfc7e704e-kube-api-access-hjscj\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:56:38.247340 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:38.247290    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-ldlf7" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": context deadline exceeded"
May 25 13:56:38.428301 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:38.428252    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:38.429537 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:38.429499    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 25 13:56:38.496241 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:38.496207    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" path="/var/lib/kubelet/pods/a2d76a4f-4432-4607-a977-6b6bfc7e704e/volumes"
May 25 13:56:39.431951 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:39.431907    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 25 13:56:44.436339 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:44.436301    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:56:44.437058 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:44.437021    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 25 13:56:54.437422 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:56:54.437393    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:57:06.558771 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.558736    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9_53d73991-307c-44a3-93ac-3dc15d8f86ea/kserve-container/0.log"
May 25 13:57:06.708747 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.708700    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"]
May 25 13:57:06.709082 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.709029    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kserve-container" containerID="cri-o://93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45" gracePeriod=30
May 25 13:57:06.709209 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.709109    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kube-rbac-proxy" containerID="cri-o://255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f" gracePeriod=30
May 25 13:57:06.790097 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790060    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"]
May 25 13:57:06.790368 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790354    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kserve-container"
May 25 13:57:06.790368 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790369    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kserve-container"
May 25 13:57:06.790519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790380    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kube-rbac-proxy"
May 25 13:57:06.790519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790385    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kube-rbac-proxy"
May 25 13:57:06.790519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790402    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="storage-initializer"
May 25 13:57:06.790519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790408    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="storage-initializer"
May 25 13:57:06.790519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790451    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kube-rbac-proxy"
May 25 13:57:06.790519 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.790459    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2d76a4f-4432-4607-a977-6b6bfc7e704e" containerName="kserve-container"
May 25 13:57:06.792462 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.792445    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:06.794935 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.794915    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 13:57:06.794999 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.794951    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
May 25 13:57:06.805863 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.805839    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"]
May 25 13:57:06.939017 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.938978    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74235c9c-85b3-44ab-811a-b6d67db10ef8-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:06.939017 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.939018    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74235c9c-85b3-44ab-811a-b6d67db10ef8-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:06.939230 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.939104    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74235c9c-85b3-44ab-811a-b6d67db10ef8-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:06.939230 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:06.939135    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrfh\" (UniqueName: \"kubernetes.io/projected/74235c9c-85b3-44ab-811a-b6d67db10ef8-kube-api-access-hfrfh\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.040377 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.040326    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74235c9c-85b3-44ab-811a-b6d67db10ef8-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.040377 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.040383    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrfh\" (UniqueName: \"kubernetes.io/projected/74235c9c-85b3-44ab-811a-b6d67db10ef8-kube-api-access-hfrfh\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.040648 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.040430    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74235c9c-85b3-44ab-811a-b6d67db10ef8-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.040648 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.040457    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74235c9c-85b3-44ab-811a-b6d67db10ef8-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.040792 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.040770    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74235c9c-85b3-44ab-811a-b6d67db10ef8-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.041091 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.041068    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74235c9c-85b3-44ab-811a-b6d67db10ef8-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.044146 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.044120    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74235c9c-85b3-44ab-811a-b6d67db10ef8-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.048088 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.048067    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrfh\" (UniqueName: \"kubernetes.io/projected/74235c9c-85b3-44ab-811a-b6d67db10ef8-kube-api-access-hfrfh\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.103229 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.103125    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:07.235917 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.235865    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"]
May 25 13:57:07.238695 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:57:07.238667    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74235c9c_85b3_44ab_811a_b6d67db10ef8.slice/crio-4221f4c3d3ddf9793a42722555265b5a714ec7bdbaa19258311bd12a2fdb8844 WatchSource:0}: Error finding container 4221f4c3d3ddf9793a42722555265b5a714ec7bdbaa19258311bd12a2fdb8844: Status 404 returned error can't find the container with id 4221f4c3d3ddf9793a42722555265b5a714ec7bdbaa19258311bd12a2fdb8844
May 25 13:57:07.512390 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.512350    2549 generic.go:358] "Generic (PLEG): container finished" podID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerID="255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f" exitCode=2
May 25 13:57:07.512390 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.512383    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" event={"ID":"53d73991-307c-44a3-93ac-3dc15d8f86ea","Type":"ContainerDied","Data":"255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f"}
May 25 13:57:07.513870 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.513845    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" event={"ID":"74235c9c-85b3-44ab-811a-b6d67db10ef8","Type":"ContainerStarted","Data":"b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce"}
May 25 13:57:07.514005 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.513877    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" event={"ID":"74235c9c-85b3-44ab-811a-b6d67db10ef8","Type":"ContainerStarted","Data":"4221f4c3d3ddf9793a42722555265b5a714ec7bdbaa19258311bd12a2fdb8844"}
May 25 13:57:07.747194 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.747170    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:57:07.847662 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.847563    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d73991-307c-44a3-93ac-3dc15d8f86ea-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"53d73991-307c-44a3-93ac-3dc15d8f86ea\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") "
May 25 13:57:07.847662 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.847608    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls\") pod \"53d73991-307c-44a3-93ac-3dc15d8f86ea\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") "
May 25 13:57:07.847860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.847670    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53d73991-307c-44a3-93ac-3dc15d8f86ea-kserve-provision-location\") pod \"53d73991-307c-44a3-93ac-3dc15d8f86ea\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") "
May 25 13:57:07.847860 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.847773    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv6zn\" (UniqueName: \"kubernetes.io/projected/53d73991-307c-44a3-93ac-3dc15d8f86ea-kube-api-access-dv6zn\") pod \"53d73991-307c-44a3-93ac-3dc15d8f86ea\" (UID: \"53d73991-307c-44a3-93ac-3dc15d8f86ea\") "
May 25 13:57:07.848033 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.847996    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d73991-307c-44a3-93ac-3dc15d8f86ea-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "53d73991-307c-44a3-93ac-3dc15d8f86ea" (UID: "53d73991-307c-44a3-93ac-3dc15d8f86ea"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:57:07.849850 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.849822    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53d73991-307c-44a3-93ac-3dc15d8f86ea" (UID: "53d73991-307c-44a3-93ac-3dc15d8f86ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:57:07.849986 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.849961    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d73991-307c-44a3-93ac-3dc15d8f86ea-kube-api-access-dv6zn" (OuterVolumeSpecName: "kube-api-access-dv6zn") pod "53d73991-307c-44a3-93ac-3dc15d8f86ea" (UID: "53d73991-307c-44a3-93ac-3dc15d8f86ea"). InnerVolumeSpecName "kube-api-access-dv6zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:57:07.872754 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.872710    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d73991-307c-44a3-93ac-3dc15d8f86ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53d73991-307c-44a3-93ac-3dc15d8f86ea" (UID: "53d73991-307c-44a3-93ac-3dc15d8f86ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:57:07.948421 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.948384    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53d73991-307c-44a3-93ac-3dc15d8f86ea-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:57:07.948421 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.948414    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6zn\" (UniqueName: \"kubernetes.io/projected/53d73991-307c-44a3-93ac-3dc15d8f86ea-kube-api-access-dv6zn\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:57:07.948421 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.948425    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d73991-307c-44a3-93ac-3dc15d8f86ea-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:57:07.948657 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:07.948435    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d73991-307c-44a3-93ac-3dc15d8f86ea-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:57:08.517887 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.517849    2549 generic.go:358] "Generic (PLEG): container finished" podID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerID="93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45" exitCode=0
May 25 13:57:08.518040 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.517939    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"
May 25 13:57:08.518040 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.517932    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" event={"ID":"53d73991-307c-44a3-93ac-3dc15d8f86ea","Type":"ContainerDied","Data":"93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45"}
May 25 13:57:08.518122 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.518044    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9" event={"ID":"53d73991-307c-44a3-93ac-3dc15d8f86ea","Type":"ContainerDied","Data":"fd2795cd4384b6b67641d2bafbbe7c06da71dc581d600a09a5051bf095b11d2d"}
May 25 13:57:08.518122 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.518060    2549 scope.go:117] "RemoveContainer" containerID="255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f"
May 25 13:57:08.525829 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.525800    2549 scope.go:117] "RemoveContainer" containerID="93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45"
May 25 13:57:08.533003 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.532980    2549 scope.go:117] "RemoveContainer" containerID="7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006"
May 25 13:57:08.538197 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.535663    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"]
May 25 13:57:08.540921 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.540901    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-28vn9"]
May 25 13:57:08.544699 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.544663    2549 scope.go:117] "RemoveContainer" containerID="255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f"
May 25 13:57:08.544972 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:57:08.544953    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f\": container with ID starting with 255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f not found: ID does not exist" containerID="255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f"
May 25 13:57:08.545025 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.544980    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f"} err="failed to get container status \"255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f\": rpc error: code = NotFound desc = could not find container \"255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f\": container with ID starting with 255e928ddcff147834da6c3e19889d98599db59bc973f38d34c603f467bb973f not found: ID does not exist"
May 25 13:57:08.545025 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.545001    2549 scope.go:117] "RemoveContainer" containerID="93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45"
May 25 13:57:08.545226 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:57:08.545211    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45\": container with ID starting with 93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45 not found: ID does not exist" containerID="93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45"
May 25 13:57:08.545291 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.545231    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45"} err="failed to get container status \"93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45\": rpc error: code = NotFound desc = could not find container \"93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45\": container with ID starting with 93957277e5d07bd3a662793dc1a75f2b8a8fdb7828f5e30f9fd093d4946c5e45 not found: ID does not exist"
May 25 13:57:08.545291 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.545243    2549 scope.go:117] "RemoveContainer" containerID="7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006"
May 25 13:57:08.545542 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:57:08.545519    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006\": container with ID starting with 7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006 not found: ID does not exist" containerID="7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006"
May 25 13:57:08.545593 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:08.545547    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006"} err="failed to get container status \"7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006\": rpc error: code = NotFound desc = could not find container \"7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006\": container with ID starting with 7d3f1dd7a82248cfc74a4fe2887d507c4749396bd01e90ed9f6980d0648ab006 not found: ID does not exist"
May 25 13:57:10.496416 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:10.496385    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" path="/var/lib/kubelet/pods/53d73991-307c-44a3-93ac-3dc15d8f86ea/volumes"
May 25 13:57:11.530206 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:11.530178    2549 generic.go:358] "Generic (PLEG): container finished" podID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerID="b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce" exitCode=0
May 25 13:57:11.530673 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:11.530227    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" event={"ID":"74235c9c-85b3-44ab-811a-b6d67db10ef8","Type":"ContainerDied","Data":"b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce"}
May 25 13:57:12.534631 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:12.534591    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" event={"ID":"74235c9c-85b3-44ab-811a-b6d67db10ef8","Type":"ContainerStarted","Data":"d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d"}
May 25 13:57:12.535043 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:12.534640    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" event={"ID":"74235c9c-85b3-44ab-811a-b6d67db10ef8","Type":"ContainerStarted","Data":"2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1"}
May 25 13:57:12.535043 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:12.534979    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:12.535043 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:12.535010    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:12.577651 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:12.577598    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" podStartSLOduration=6.5775819559999995 podStartE2EDuration="6.577581956s" podCreationTimestamp="2026-05-25 13:57:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:57:12.574425955 +0000 UTC m=+3182.613183091" watchObservedRunningTime="2026-05-25 13:57:12.577581956 +0000 UTC m=+3182.616339081"
May 25 13:57:18.543102 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:18.543068    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:57:48.592473 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:48.592420    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 13:57:58.545851 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:57:58.545817    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:58:06.879645 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.879602    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"]
May 25 13:58:06.880223 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.879916    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kserve-container" containerID="cri-o://2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1" gracePeriod=30
May 25 13:58:06.880223 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.879966    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kube-rbac-proxy" containerID="cri-o://d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d" gracePeriod=30
May 25 13:58:06.937077 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937039    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"]
May 25 13:58:06.937331 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937318    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kube-rbac-proxy"
May 25 13:58:06.937387 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937332    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kube-rbac-proxy"
May 25 13:58:06.937387 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937349    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kserve-container"
May 25 13:58:06.937387 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937355    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kserve-container"
May 25 13:58:06.937387 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937364    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="storage-initializer"
May 25 13:58:06.937387 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937370    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="storage-initializer"
May 25 13:58:06.937546 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937430    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kube-rbac-proxy"
May 25 13:58:06.937546 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.937439    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="53d73991-307c-44a3-93ac-3dc15d8f86ea" containerName="kserve-container"
May 25 13:58:06.939246 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.939230    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:06.941224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.941188    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
May 25 13:58:06.941478 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.941456    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
May 25 13:58:06.950213 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:06.950189    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"]
May 25 13:58:07.006528 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.006504    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f902ff6e-e7d4-465c-aaa2-04123e19d777-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.006667 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.006544    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.006667 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.006637    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgxgc\" (UniqueName: \"kubernetes.io/projected/f902ff6e-e7d4-465c-aaa2-04123e19d777-kube-api-access-zgxgc\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.006750 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.006689    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f902ff6e-e7d4-465c-aaa2-04123e19d777-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.107853 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.107817    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f902ff6e-e7d4-465c-aaa2-04123e19d777-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.108009 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.107862    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.108009 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.107906    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxgc\" (UniqueName: \"kubernetes.io/projected/f902ff6e-e7d4-465c-aaa2-04123e19d777-kube-api-access-zgxgc\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.108009 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.107922    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f902ff6e-e7d4-465c-aaa2-04123e19d777-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.108188 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:58:07.108057    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
May 25 13:58:07.108188 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:58:07.108146    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls podName:f902ff6e-e7d4-465c-aaa2-04123e19d777 nodeName:}" failed. No retries permitted until 2026-05-25 13:58:07.608122165 +0000 UTC m=+3237.646879279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls") pod "isvc-sklearn-v2-predictor-9df9c96c7-66mrb" (UID: "f902ff6e-e7d4-465c-aaa2-04123e19d777") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
May 25 13:58:07.108437 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.108414    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f902ff6e-e7d4-465c-aaa2-04123e19d777-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.108522 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.108503    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f902ff6e-e7d4-465c-aaa2-04123e19d777-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.116624 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.116598    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxgc\" (UniqueName: \"kubernetes.io/projected/f902ff6e-e7d4-465c-aaa2-04123e19d777-kube-api-access-zgxgc\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.612125 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.612088    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.614923 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.614894    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-66mrb\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.685420 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.685385    2549 generic.go:358] "Generic (PLEG): container finished" podID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerID="d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d" exitCode=2
May 25 13:58:07.685599 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.685462    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" event={"ID":"74235c9c-85b3-44ab-811a-b6d67db10ef8","Type":"ContainerDied","Data":"d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d"}
May 25 13:58:07.849445 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.849390    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:07.981959 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.981876    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"]
May 25 13:58:07.984389 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:58:07.984346    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf902ff6e_e7d4_465c_aaa2_04123e19d777.slice/crio-683b381d2ea2849bd1af9e6fc6540d15d7136c551c09d17cced85e878de14fbd WatchSource:0}: Error finding container 683b381d2ea2849bd1af9e6fc6540d15d7136c551c09d17cced85e878de14fbd: Status 404 returned error can't find the container with id 683b381d2ea2849bd1af9e6fc6540d15d7136c551c09d17cced85e878de14fbd
May 25 13:58:07.986395 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:07.986374    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:58:08.537847 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:08.537802    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 25 13:58:08.690253 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:08.690219    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" event={"ID":"f902ff6e-e7d4-465c-aaa2-04123e19d777","Type":"ContainerStarted","Data":"ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e"}
May 25 13:58:08.690253 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:08.690257    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" event={"ID":"f902ff6e-e7d4-465c-aaa2-04123e19d777","Type":"ContainerStarted","Data":"683b381d2ea2849bd1af9e6fc6540d15d7136c551c09d17cced85e878de14fbd"}
May 25 13:58:09.585475 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:09.585422    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.47:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 25 13:58:11.701631 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:11.701594    2549 generic.go:358] "Generic (PLEG): container finished" podID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerID="ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e" exitCode=0
May 25 13:58:11.701993 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:11.701645    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" event={"ID":"f902ff6e-e7d4-465c-aaa2-04123e19d777","Type":"ContainerDied","Data":"ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e"}
May 25 13:58:12.706521 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:12.706489    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" event={"ID":"f902ff6e-e7d4-465c-aaa2-04123e19d777","Type":"ContainerStarted","Data":"12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3"}
May 25 13:58:12.706930 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:12.706530    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" event={"ID":"f902ff6e-e7d4-465c-aaa2-04123e19d777","Type":"ContainerStarted","Data":"f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d"}
May 25 13:58:12.706930 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:12.706831    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:12.707013 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:12.706969    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:12.708292 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:12.708248    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:58:12.727509 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:12.727466    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podStartSLOduration=6.727452101 podStartE2EDuration="6.727452101s" podCreationTimestamp="2026-05-25 13:58:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:58:12.726749651 +0000 UTC m=+3242.765506786" watchObservedRunningTime="2026-05-25 13:58:12.727452101 +0000 UTC m=+3242.766209224"
May 25 13:58:13.537635 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:13.537592    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 25 13:58:13.709064 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:13.709016    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:58:14.612659 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.612635    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:58:14.713572 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.713480    2549 generic.go:358] "Generic (PLEG): container finished" podID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerID="2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1" exitCode=0
May 25 13:58:14.713572 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.713564    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"
May 25 13:58:14.714029 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.713568    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" event={"ID":"74235c9c-85b3-44ab-811a-b6d67db10ef8","Type":"ContainerDied","Data":"2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1"}
May 25 13:58:14.714029 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.713607    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc" event={"ID":"74235c9c-85b3-44ab-811a-b6d67db10ef8","Type":"ContainerDied","Data":"4221f4c3d3ddf9793a42722555265b5a714ec7bdbaa19258311bd12a2fdb8844"}
May 25 13:58:14.714029 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.713628    2549 scope.go:117] "RemoveContainer" containerID="d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d"
May 25 13:58:14.720993 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.720971    2549 scope.go:117] "RemoveContainer" containerID="2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1"
May 25 13:58:14.727859 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.727838    2549 scope.go:117] "RemoveContainer" containerID="b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce"
May 25 13:58:14.734641 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.734623    2549 scope.go:117] "RemoveContainer" containerID="d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d"
May 25 13:58:14.734908 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:58:14.734889    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d\": container with ID starting with d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d not found: ID does not exist" containerID="d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d"
May 25 13:58:14.734966 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.734920    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d"} err="failed to get container status \"d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d\": rpc error: code = NotFound desc = could not find container \"d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d\": container with ID starting with d887697da4f1310f343850f43151f99dcb48ec804f1da75427b92fe05d38668d not found: ID does not exist"
May 25 13:58:14.734966 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.734937    2549 scope.go:117] "RemoveContainer" containerID="2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1"
May 25 13:58:14.735145 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:58:14.735129    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1\": container with ID starting with 2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1 not found: ID does not exist" containerID="2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1"
May 25 13:58:14.735195 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.735151    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1"} err="failed to get container status \"2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1\": rpc error: code = NotFound desc = could not find container \"2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1\": container with ID starting with 2030313a24f94b8e4a3df3fbdaf36aa52aa9831aeb77a775a50ebf731df8bff1 not found: ID does not exist"
May 25 13:58:14.735195 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.735166    2549 scope.go:117] "RemoveContainer" containerID="b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce"
May 25 13:58:14.735364 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:58:14.735349    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce\": container with ID starting with b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce not found: ID does not exist" containerID="b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce"
May 25 13:58:14.735419 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.735367    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce"} err="failed to get container status \"b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce\": rpc error: code = NotFound desc = could not find container \"b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce\": container with ID starting with b1562b3dece7869e0e78695e620ab8d094265769041141d023044da26f87a5ce not found: ID does not exist"
May 25 13:58:14.767313 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.767279    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfrfh\" (UniqueName: \"kubernetes.io/projected/74235c9c-85b3-44ab-811a-b6d67db10ef8-kube-api-access-hfrfh\") pod \"74235c9c-85b3-44ab-811a-b6d67db10ef8\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") "
May 25 13:58:14.767443 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.767330    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74235c9c-85b3-44ab-811a-b6d67db10ef8-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"74235c9c-85b3-44ab-811a-b6d67db10ef8\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") "
May 25 13:58:14.767443 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.767380    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74235c9c-85b3-44ab-811a-b6d67db10ef8-kserve-provision-location\") pod \"74235c9c-85b3-44ab-811a-b6d67db10ef8\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") "
May 25 13:58:14.767443 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.767418    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74235c9c-85b3-44ab-811a-b6d67db10ef8-proxy-tls\") pod \"74235c9c-85b3-44ab-811a-b6d67db10ef8\" (UID: \"74235c9c-85b3-44ab-811a-b6d67db10ef8\") "
May 25 13:58:14.767691 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.767668    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74235c9c-85b3-44ab-811a-b6d67db10ef8-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "74235c9c-85b3-44ab-811a-b6d67db10ef8" (UID: "74235c9c-85b3-44ab-811a-b6d67db10ef8"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:58:14.767763 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.767727    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74235c9c-85b3-44ab-811a-b6d67db10ef8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74235c9c-85b3-44ab-811a-b6d67db10ef8" (UID: "74235c9c-85b3-44ab-811a-b6d67db10ef8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:58:14.769502 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.769474    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74235c9c-85b3-44ab-811a-b6d67db10ef8-kube-api-access-hfrfh" (OuterVolumeSpecName: "kube-api-access-hfrfh") pod "74235c9c-85b3-44ab-811a-b6d67db10ef8" (UID: "74235c9c-85b3-44ab-811a-b6d67db10ef8"). InnerVolumeSpecName "kube-api-access-hfrfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:58:14.769619 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.769531    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74235c9c-85b3-44ab-811a-b6d67db10ef8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74235c9c-85b3-44ab-811a-b6d67db10ef8" (UID: "74235c9c-85b3-44ab-811a-b6d67db10ef8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:58:14.868893 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.868854    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74235c9c-85b3-44ab-811a-b6d67db10ef8-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:58:14.868893 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.868887    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74235c9c-85b3-44ab-811a-b6d67db10ef8-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:58:14.868893 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.868898    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfrfh\" (UniqueName: \"kubernetes.io/projected/74235c9c-85b3-44ab-811a-b6d67db10ef8-kube-api-access-hfrfh\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:58:14.869132 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:14.868908    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74235c9c-85b3-44ab-811a-b6d67db10ef8-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:58:15.034380 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:15.034348    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"]
May 25 13:58:15.038259 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:15.038228    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-9d7rc"]
May 25 13:58:16.495423 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:16.495387    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" path="/var/lib/kubelet/pods/74235c9c-85b3-44ab-811a-b6d67db10ef8/volumes"
May 25 13:58:18.715113 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:18.715082    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:58:18.715728 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:18.715699    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:58:28.716224 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:28.716176    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:58:38.716090 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:38.716038    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:58:48.716420 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:48.716376    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:58:58.716388 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:58:58.716347    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:59:08.716190 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:08.716149    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:59:11.199401 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:11.199369    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:59:11.204819 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:11.204797    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 13:59:18.716586 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:18.716500    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:59:27.181751 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.181711    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"]
May 25 13:59:27.182179 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.182036    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" containerID="cri-o://f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d" gracePeriod=30
May 25 13:59:27.182179 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.182065    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kube-rbac-proxy" containerID="cri-o://12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3" gracePeriod=30
May 25 13:59:27.265636 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.265594    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"]
May 25 13:59:27.265939 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.265922    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kserve-container"
May 25 13:59:27.266025 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.265942    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kserve-container"
May 25 13:59:27.266025 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.265960    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="storage-initializer"
May 25 13:59:27.266025 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.265968    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="storage-initializer"
May 25 13:59:27.266025 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.265977    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kube-rbac-proxy"
May 25 13:59:27.266025 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.265986    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kube-rbac-proxy"
May 25 13:59:27.266310 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.266050    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kube-rbac-proxy"
May 25 13:59:27.266310 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.266063    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="74235c9c-85b3-44ab-811a-b6d67db10ef8" containerName="kserve-container"
May 25 13:59:27.268987 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.268965    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.271174 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.271145    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
May 25 13:59:27.271324 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.271146    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
May 25 13:59:27.277785 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.277753    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"]
May 25 13:59:27.409548 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.409497    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17992003-ad90-44b4-9d86-c57504d7b601-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.409746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.409556    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17992003-ad90-44b4-9d86-c57504d7b601-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.409746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.409585    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17992003-ad90-44b4-9d86-c57504d7b601-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.409746 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.409607    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprgx\" (UniqueName: \"kubernetes.io/projected/17992003-ad90-44b4-9d86-c57504d7b601-kube-api-access-tprgx\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.510702 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.510611    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17992003-ad90-44b4-9d86-c57504d7b601-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.510702 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.510656    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17992003-ad90-44b4-9d86-c57504d7b601-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.510926 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.510779    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17992003-ad90-44b4-9d86-c57504d7b601-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.510926 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.510818    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tprgx\" (UniqueName: \"kubernetes.io/projected/17992003-ad90-44b4-9d86-c57504d7b601-kube-api-access-tprgx\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.511080 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.511060    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17992003-ad90-44b4-9d86-c57504d7b601-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.511253 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.511235    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17992003-ad90-44b4-9d86-c57504d7b601-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.513230 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.513210    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17992003-ad90-44b4-9d86-c57504d7b601-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.519914 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.519890    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprgx\" (UniqueName: \"kubernetes.io/projected/17992003-ad90-44b4-9d86-c57504d7b601-kube-api-access-tprgx\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.579970 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.579924    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:27.709542 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.709494    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"]
May 25 13:59:27.712834 ip-10-0-129-55 kubenswrapper[2549]: W0525 13:59:27.712803    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17992003_ad90_44b4_9d86_c57504d7b601.slice/crio-0c20de4da8b2476783f52cfd7fd71693954edf708a681a3299e04946e742eef2 WatchSource:0}: Error finding container 0c20de4da8b2476783f52cfd7fd71693954edf708a681a3299e04946e742eef2: Status 404 returned error can't find the container with id 0c20de4da8b2476783f52cfd7fd71693954edf708a681a3299e04946e742eef2
May 25 13:59:27.923223 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.923190    2549 generic.go:358] "Generic (PLEG): container finished" podID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerID="12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3" exitCode=2
May 25 13:59:27.923437 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.923262    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" event={"ID":"f902ff6e-e7d4-465c-aaa2-04123e19d777","Type":"ContainerDied","Data":"12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3"}
May 25 13:59:27.924601 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.924579    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" event={"ID":"17992003-ad90-44b4-9d86-c57504d7b601","Type":"ContainerStarted","Data":"a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2"}
May 25 13:59:27.924601 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:27.924603    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" event={"ID":"17992003-ad90-44b4-9d86-c57504d7b601","Type":"ContainerStarted","Data":"0c20de4da8b2476783f52cfd7fd71693954edf708a681a3299e04946e742eef2"}
May 25 13:59:28.710337 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:28.710265    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 25 13:59:28.716569 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:28.716533    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 13:59:31.723639 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.723614    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:59:31.842059 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.841965    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f902ff6e-e7d4-465c-aaa2-04123e19d777-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"f902ff6e-e7d4-465c-aaa2-04123e19d777\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") "
May 25 13:59:31.842059 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.842027    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f902ff6e-e7d4-465c-aaa2-04123e19d777-kserve-provision-location\") pod \"f902ff6e-e7d4-465c-aaa2-04123e19d777\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") "
May 25 13:59:31.842059 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.842061    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgxgc\" (UniqueName: \"kubernetes.io/projected/f902ff6e-e7d4-465c-aaa2-04123e19d777-kube-api-access-zgxgc\") pod \"f902ff6e-e7d4-465c-aaa2-04123e19d777\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") "
May 25 13:59:31.842325 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.842087    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls\") pod \"f902ff6e-e7d4-465c-aaa2-04123e19d777\" (UID: \"f902ff6e-e7d4-465c-aaa2-04123e19d777\") "
May 25 13:59:31.842393 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.842369    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f902ff6e-e7d4-465c-aaa2-04123e19d777-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "f902ff6e-e7d4-465c-aaa2-04123e19d777" (UID: "f902ff6e-e7d4-465c-aaa2-04123e19d777"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:59:31.842454 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.842429    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f902ff6e-e7d4-465c-aaa2-04123e19d777-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f902ff6e-e7d4-465c-aaa2-04123e19d777" (UID: "f902ff6e-e7d4-465c-aaa2-04123e19d777"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:59:31.844351 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.844324    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f902ff6e-e7d4-465c-aaa2-04123e19d777" (UID: "f902ff6e-e7d4-465c-aaa2-04123e19d777"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:59:31.844444 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.844343    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f902ff6e-e7d4-465c-aaa2-04123e19d777-kube-api-access-zgxgc" (OuterVolumeSpecName: "kube-api-access-zgxgc") pod "f902ff6e-e7d4-465c-aaa2-04123e19d777" (UID: "f902ff6e-e7d4-465c-aaa2-04123e19d777"). InnerVolumeSpecName "kube-api-access-zgxgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:59:31.936345 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.936311    2549 generic.go:358] "Generic (PLEG): container finished" podID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerID="f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d" exitCode=0
May 25 13:59:31.936544 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.936399    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"
May 25 13:59:31.936544 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.936402    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" event={"ID":"f902ff6e-e7d4-465c-aaa2-04123e19d777","Type":"ContainerDied","Data":"f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d"}
May 25 13:59:31.936544 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.936443    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb" event={"ID":"f902ff6e-e7d4-465c-aaa2-04123e19d777","Type":"ContainerDied","Data":"683b381d2ea2849bd1af9e6fc6540d15d7136c551c09d17cced85e878de14fbd"}
May 25 13:59:31.936544 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.936460    2549 scope.go:117] "RemoveContainer" containerID="12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3"
May 25 13:59:31.937881 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.937860    2549 generic.go:358] "Generic (PLEG): container finished" podID="17992003-ad90-44b4-9d86-c57504d7b601" containerID="a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2" exitCode=0
May 25 13:59:31.938000 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.937923    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" event={"ID":"17992003-ad90-44b4-9d86-c57504d7b601","Type":"ContainerDied","Data":"a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2"}
May 25 13:59:31.943127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.943042    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgxgc\" (UniqueName: \"kubernetes.io/projected/f902ff6e-e7d4-465c-aaa2-04123e19d777-kube-api-access-zgxgc\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:59:31.943127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.943068    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f902ff6e-e7d4-465c-aaa2-04123e19d777-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:59:31.943127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.943084    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f902ff6e-e7d4-465c-aaa2-04123e19d777-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:59:31.943127 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.943099    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f902ff6e-e7d4-465c-aaa2-04123e19d777-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 13:59:31.945185 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.945162    2549 scope.go:117] "RemoveContainer" containerID="f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d"
May 25 13:59:31.952320 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.952302    2549 scope.go:117] "RemoveContainer" containerID="ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e"
May 25 13:59:31.960064 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.960043    2549 scope.go:117] "RemoveContainer" containerID="12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3"
May 25 13:59:31.960552 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:59:31.960522    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3\": container with ID starting with 12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3 not found: ID does not exist" containerID="12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3"
May 25 13:59:31.960642 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.960564    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3"} err="failed to get container status \"12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3\": rpc error: code = NotFound desc = could not find container \"12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3\": container with ID starting with 12f4cbcd7b1dc189f52ad96a0c843bfc352336ba73e86b9a619eaa5fc24575e3 not found: ID does not exist"
May 25 13:59:31.960642 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.960587    2549 scope.go:117] "RemoveContainer" containerID="f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d"
May 25 13:59:31.960891 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:59:31.960871    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d\": container with ID starting with f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d not found: ID does not exist" containerID="f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d"
May 25 13:59:31.960995 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.960897    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d"} err="failed to get container status \"f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d\": rpc error: code = NotFound desc = could not find container \"f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d\": container with ID starting with f928cb172168f28a5aa6861bd089caee45c1787d5fc2f6e7adf42daad015e01d not found: ID does not exist"
May 25 13:59:31.960995 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.960912    2549 scope.go:117] "RemoveContainer" containerID="ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e"
May 25 13:59:31.961203 ip-10-0-129-55 kubenswrapper[2549]: E0525 13:59:31.961173    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e\": container with ID starting with ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e not found: ID does not exist" containerID="ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e"
May 25 13:59:31.961261 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.961215    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e"} err="failed to get container status \"ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e\": rpc error: code = NotFound desc = could not find container \"ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e\": container with ID starting with ac16c43c8f7326766a4f3937d9819d7efb04eebd4b943ba3cdff6daaee9eff4e not found: ID does not exist"
May 25 13:59:31.972755 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.972717    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"]
May 25 13:59:31.978620 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:31.978597    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-66mrb"]
May 25 13:59:32.497751 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:32.497713    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" path="/var/lib/kubelet/pods/f902ff6e-e7d4-465c-aaa2-04123e19d777/volumes"
May 25 13:59:32.947595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:32.947559    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" event={"ID":"17992003-ad90-44b4-9d86-c57504d7b601","Type":"ContainerStarted","Data":"bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df"}
May 25 13:59:32.947595 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:32.947594    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" event={"ID":"17992003-ad90-44b4-9d86-c57504d7b601","Type":"ContainerStarted","Data":"8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5"}
May 25 13:59:32.948045 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:32.947901    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:32.966195 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:32.966144    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podStartSLOduration=5.966130435 podStartE2EDuration="5.966130435s" podCreationTimestamp="2026-05-25 13:59:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:59:32.965080392 +0000 UTC m=+3323.003837508" watchObservedRunningTime="2026-05-25 13:59:32.966130435 +0000 UTC m=+3323.004887558"
May 25 13:59:33.950583 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:33.950544    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:33.951894 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:33.951864    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 13:59:34.953495 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:34.953454    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 13:59:39.958458 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:39.958429    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 13:59:39.959054 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:39.959022    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 13:59:49.959798 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:49.959754    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 13:59:59.959376 ip-10-0-129-55 kubenswrapper[2549]: I0525 13:59:59.959334    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 14:00:09.959065 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:09.959016    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 14:00:19.959514 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:19.959475    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 14:00:29.959249 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:29.959201    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 14:00:39.959448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:39.959411    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 14:00:47.361042 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.361001    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"]
May 25 14:00:47.361537 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.361482    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" containerID="cri-o://8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5" gracePeriod=30
May 25 14:00:47.361641 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.361519    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kube-rbac-proxy" containerID="cri-o://bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df" gracePeriod=30
May 25 14:00:47.452332 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452293    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"]
May 25 14:00:47.452597 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452584    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container"
May 25 14:00:47.452670 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452598    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container"
May 25 14:00:47.452670 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452613    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kube-rbac-proxy"
May 25 14:00:47.452670 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452619    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kube-rbac-proxy"
May 25 14:00:47.452670 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452633    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="storage-initializer"
May 25 14:00:47.452670 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452639    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="storage-initializer"
May 25 14:00:47.452831 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452678    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kserve-container"
May 25 14:00:47.452831 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.452686    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="f902ff6e-e7d4-465c-aaa2-04123e19d777" containerName="kube-rbac-proxy"
May 25 14:00:47.454578 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.454555    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.456900 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.456880    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
May 25 14:00:47.457004 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.456923    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
May 25 14:00:47.464694 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.464672    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"]
May 25 14:00:47.513417 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.513381    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b88b1335-a7a0-4644-b2df-1d8de71a052e-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.513417 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.513419    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b88b1335-a7a0-4644-b2df-1d8de71a052e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.513620 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.513450    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.513620 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.513536    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdk6n\" (UniqueName: \"kubernetes.io/projected/b88b1335-a7a0-4644-b2df-1d8de71a052e-kube-api-access-fdk6n\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.614056 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.614026    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b88b1335-a7a0-4644-b2df-1d8de71a052e-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.614216 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.614071    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b88b1335-a7a0-4644-b2df-1d8de71a052e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.614216 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.614114    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.614216 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.614191    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fdk6n\" (UniqueName: \"kubernetes.io/projected/b88b1335-a7a0-4644-b2df-1d8de71a052e-kube-api-access-fdk6n\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.614448 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:00:47.614323    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
May 25 14:00:47.614448 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:00:47.614412    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls podName:b88b1335-a7a0-4644-b2df-1d8de71a052e nodeName:}" failed. No retries permitted until 2026-05-25 14:00:48.114385406 +0000 UTC m=+3398.153142528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls") pod "isvc-tensorflow-predictor-6756f669d7-tw8xz" (UID: "b88b1335-a7a0-4644-b2df-1d8de71a052e") : secret "isvc-tensorflow-predictor-serving-cert" not found
May 25 14:00:47.614652 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.614540    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b88b1335-a7a0-4644-b2df-1d8de71a052e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.614815 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.614796    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b88b1335-a7a0-4644-b2df-1d8de71a052e-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:47.623394 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:47.623343    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdk6n\" (UniqueName: \"kubernetes.io/projected/b88b1335-a7a0-4644-b2df-1d8de71a052e-kube-api-access-fdk6n\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:48.118806 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:48.118775    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:48.121314 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:48.121294    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-tw8xz\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:48.153218 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:48.153185    2549 generic.go:358] "Generic (PLEG): container finished" podID="17992003-ad90-44b4-9d86-c57504d7b601" containerID="bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df" exitCode=2
May 25 14:00:48.153410 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:48.153246    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" event={"ID":"17992003-ad90-44b4-9d86-c57504d7b601","Type":"ContainerDied","Data":"bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df"}
May 25 14:00:48.364564 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:48.364523    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:48.497091 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:48.497065    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"]
May 25 14:00:48.499486 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:00:48.499460    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88b1335_a7a0_4644_b2df_1d8de71a052e.slice/crio-d5695ca10774d3e3a92c77e5f7438ad2aabbffc60d3ee5a6c2a0e13575ea1935 WatchSource:0}: Error finding container d5695ca10774d3e3a92c77e5f7438ad2aabbffc60d3ee5a6c2a0e13575ea1935: Status 404 returned error can't find the container with id d5695ca10774d3e3a92c77e5f7438ad2aabbffc60d3ee5a6c2a0e13575ea1935
May 25 14:00:49.157373 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:49.157340    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" event={"ID":"b88b1335-a7a0-4644-b2df-1d8de71a052e","Type":"ContainerStarted","Data":"fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e"}
May 25 14:00:49.157373 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:49.157376    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" event={"ID":"b88b1335-a7a0-4644-b2df-1d8de71a052e","Type":"ContainerStarted","Data":"d5695ca10774d3e3a92c77e5f7438ad2aabbffc60d3ee5a6c2a0e13575ea1935"}
May 25 14:00:49.954051 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:49.953998    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
May 25 14:00:49.959379 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:49.959350    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 14:00:51.804689 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.804663    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 14:00:51.948415 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.948371    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17992003-ad90-44b4-9d86-c57504d7b601-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"17992003-ad90-44b4-9d86-c57504d7b601\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") "
May 25 14:00:51.948415 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.948421    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17992003-ad90-44b4-9d86-c57504d7b601-proxy-tls\") pod \"17992003-ad90-44b4-9d86-c57504d7b601\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") "
May 25 14:00:51.948674 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.948489    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17992003-ad90-44b4-9d86-c57504d7b601-kserve-provision-location\") pod \"17992003-ad90-44b4-9d86-c57504d7b601\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") "
May 25 14:00:51.948674 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.948514    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tprgx\" (UniqueName: \"kubernetes.io/projected/17992003-ad90-44b4-9d86-c57504d7b601-kube-api-access-tprgx\") pod \"17992003-ad90-44b4-9d86-c57504d7b601\" (UID: \"17992003-ad90-44b4-9d86-c57504d7b601\") "
May 25 14:00:51.948822 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.948797    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17992003-ad90-44b4-9d86-c57504d7b601-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17992003-ad90-44b4-9d86-c57504d7b601" (UID: "17992003-ad90-44b4-9d86-c57504d7b601"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:00:51.948877 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.948830    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17992003-ad90-44b4-9d86-c57504d7b601-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "17992003-ad90-44b4-9d86-c57504d7b601" (UID: "17992003-ad90-44b4-9d86-c57504d7b601"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:00:51.950673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.950648    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17992003-ad90-44b4-9d86-c57504d7b601-kube-api-access-tprgx" (OuterVolumeSpecName: "kube-api-access-tprgx") pod "17992003-ad90-44b4-9d86-c57504d7b601" (UID: "17992003-ad90-44b4-9d86-c57504d7b601"). InnerVolumeSpecName "kube-api-access-tprgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:00:51.950774 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:51.950737    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17992003-ad90-44b4-9d86-c57504d7b601-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17992003-ad90-44b4-9d86-c57504d7b601" (UID: "17992003-ad90-44b4-9d86-c57504d7b601"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:00:52.049083 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.049034    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17992003-ad90-44b4-9d86-c57504d7b601-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:00:52.049083 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.049075    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tprgx\" (UniqueName: \"kubernetes.io/projected/17992003-ad90-44b4-9d86-c57504d7b601-kube-api-access-tprgx\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:00:52.049083 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.049088    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17992003-ad90-44b4-9d86-c57504d7b601-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:00:52.049373 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.049100    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17992003-ad90-44b4-9d86-c57504d7b601-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:00:52.167489 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.167444    2549 generic.go:358] "Generic (PLEG): container finished" podID="17992003-ad90-44b4-9d86-c57504d7b601" containerID="8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5" exitCode=0
May 25 14:00:52.167673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.167499    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" event={"ID":"17992003-ad90-44b4-9d86-c57504d7b601","Type":"ContainerDied","Data":"8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5"}
May 25 14:00:52.167673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.167525    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc" event={"ID":"17992003-ad90-44b4-9d86-c57504d7b601","Type":"ContainerDied","Data":"0c20de4da8b2476783f52cfd7fd71693954edf708a681a3299e04946e742eef2"}
May 25 14:00:52.167673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.167541    2549 scope.go:117] "RemoveContainer" containerID="bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df"
May 25 14:00:52.167673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.167548    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"
May 25 14:00:52.175805 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.175779    2549 scope.go:117] "RemoveContainer" containerID="8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5"
May 25 14:00:52.182792 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.182770    2549 scope.go:117] "RemoveContainer" containerID="a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2"
May 25 14:00:52.188616 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.188588    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"]
May 25 14:00:52.190617 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.190597    2549 scope.go:117] "RemoveContainer" containerID="bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df"
May 25 14:00:52.191009 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:00:52.190956    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df\": container with ID starting with bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df not found: ID does not exist" containerID="bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df"
May 25 14:00:52.191109 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.190998    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df"} err="failed to get container status \"bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df\": rpc error: code = NotFound desc = could not find container \"bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df\": container with ID starting with bbb2de0c89520bbb1052e0ffabe6e1909e22fc9a79e31abff4dcb26e3b8fd0df not found: ID does not exist"
May 25 14:00:52.191109 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.191024    2549 scope.go:117] "RemoveContainer" containerID="8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5"
May 25 14:00:52.192338 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:00:52.192313    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5\": container with ID starting with 8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5 not found: ID does not exist" containerID="8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5"
May 25 14:00:52.192428 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.192348    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5"} err="failed to get container status \"8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5\": rpc error: code = NotFound desc = could not find container \"8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5\": container with ID starting with 8c7139d41bfaed64fc2c53781f1fe2284192a28dd53dfaea490a36deab44ddd5 not found: ID does not exist"
May 25 14:00:52.192428 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.192373    2549 scope.go:117] "RemoveContainer" containerID="a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2"
May 25 14:00:52.192662 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.192642    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-rx8tc"]
May 25 14:00:52.192728 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:00:52.192655    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2\": container with ID starting with a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2 not found: ID does not exist" containerID="a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2"
May 25 14:00:52.192728 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.192701    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2"} err="failed to get container status \"a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2\": rpc error: code = NotFound desc = could not find container \"a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2\": container with ID starting with a0f8b2952720ad105c8828f34314aca1121c38b15e8ce1c4c0cbaec2acd508e2 not found: ID does not exist"
May 25 14:00:52.495743 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:52.495697    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17992003-ad90-44b4-9d86-c57504d7b601" path="/var/lib/kubelet/pods/17992003-ad90-44b4-9d86-c57504d7b601/volumes"
May 25 14:00:53.171197 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:53.171163    2549 generic.go:358] "Generic (PLEG): container finished" podID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerID="fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e" exitCode=0
May 25 14:00:53.171600 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:53.171238    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" event={"ID":"b88b1335-a7a0-4644-b2df-1d8de71a052e","Type":"ContainerDied","Data":"fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e"}
May 25 14:00:57.186519 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:57.186424    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" event={"ID":"b88b1335-a7a0-4644-b2df-1d8de71a052e","Type":"ContainerStarted","Data":"cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2"}
May 25 14:00:57.186519 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:57.186464    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" event={"ID":"b88b1335-a7a0-4644-b2df-1d8de71a052e","Type":"ContainerStarted","Data":"de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3"}
May 25 14:00:57.186953 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:57.186753    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:57.186953 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:57.186878    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:00:57.188110 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:57.188083    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 25 14:00:57.203154 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:57.203108    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podStartSLOduration=6.473529549 podStartE2EDuration="10.203095776s" podCreationTimestamp="2026-05-25 14:00:47 +0000 UTC" firstStartedPulling="2026-05-25 14:00:53.172451326 +0000 UTC m=+3403.211208429" lastFinishedPulling="2026-05-25 14:00:56.902017552 +0000 UTC m=+3406.940774656" observedRunningTime="2026-05-25 14:00:57.202834442 +0000 UTC m=+3407.241591577" watchObservedRunningTime="2026-05-25 14:00:57.203095776 +0000 UTC m=+3407.241852900"
May 25 14:00:58.189824 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:00:58.189776    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 25 14:01:03.194489 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:03.194462    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:01:03.194996 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:03.194969    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 25 14:01:13.195101 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:13.195069    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:01:28.332248 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.332203    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"]
May 25 14:01:28.332649 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.332563    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kserve-container" containerID="cri-o://de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3" gracePeriod=30
May 25 14:01:28.332722 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.332585    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy" containerID="cri-o://cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2" gracePeriod=30
May 25 14:01:28.437257 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437219    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"]
May 25 14:01:28.437542 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437527    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container"
May 25 14:01:28.437584 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437545    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container"
May 25 14:01:28.437584 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437567    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="storage-initializer"
May 25 14:01:28.437584 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437577    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="storage-initializer"
May 25 14:01:28.437677 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437591    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kube-rbac-proxy"
May 25 14:01:28.437677 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437597    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kube-rbac-proxy"
May 25 14:01:28.437677 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437640    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kube-rbac-proxy"
May 25 14:01:28.437677 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.437654    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="17992003-ad90-44b4-9d86-c57504d7b601" containerName="kserve-container"
May 25 14:01:28.440310 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.440290    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.442544 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.442522    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
May 25 14:01:28.442737 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.442723    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
May 25 14:01:28.450150 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.450124    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"]
May 25 14:01:28.537899 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.537863    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwztj\" (UniqueName: \"kubernetes.io/projected/47ce242c-3874-4d0c-ba59-f71641401374-kube-api-access-xwztj\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.537899 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.537901    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47ce242c-3874-4d0c-ba59-f71641401374-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.538104 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.537971    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ce242c-3874-4d0c-ba59-f71641401374-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.538104 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.537990    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ce242c-3874-4d0c-ba59-f71641401374-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.638352 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.638320    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ce242c-3874-4d0c-ba59-f71641401374-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.638517 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.638361    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ce242c-3874-4d0c-ba59-f71641401374-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.638517 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.638389    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwztj\" (UniqueName: \"kubernetes.io/projected/47ce242c-3874-4d0c-ba59-f71641401374-kube-api-access-xwztj\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.638517 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.638414    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47ce242c-3874-4d0c-ba59-f71641401374-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.638958 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.638930    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47ce242c-3874-4d0c-ba59-f71641401374-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.639203 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.639153    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ce242c-3874-4d0c-ba59-f71641401374-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.640890 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.640870    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ce242c-3874-4d0c-ba59-f71641401374-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.646240 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.646214    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwztj\" (UniqueName: \"kubernetes.io/projected/47ce242c-3874-4d0c-ba59-f71641401374-kube-api-access-xwztj\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.751873 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.751831    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:28.878427 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:28.878358    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"]
May 25 14:01:28.880914 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:01:28.880885    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ce242c_3874_4d0c_ba59_f71641401374.slice/crio-0b7e8b57b521227a740b5dcb544034831c8a39c5b28f248d0a39b59c44df42a4 WatchSource:0}: Error finding container 0b7e8b57b521227a740b5dcb544034831c8a39c5b28f248d0a39b59c44df42a4: Status 404 returned error can't find the container with id 0b7e8b57b521227a740b5dcb544034831c8a39c5b28f248d0a39b59c44df42a4
May 25 14:01:29.277841 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:29.277746    2549 generic.go:358] "Generic (PLEG): container finished" podID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerID="cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2" exitCode=2
May 25 14:01:29.277841 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:29.277816    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" event={"ID":"b88b1335-a7a0-4644-b2df-1d8de71a052e","Type":"ContainerDied","Data":"cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2"}
May 25 14:01:29.279185 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:29.279160    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" event={"ID":"47ce242c-3874-4d0c-ba59-f71641401374","Type":"ContainerStarted","Data":"9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244"}
May 25 14:01:29.279377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:29.279193    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" event={"ID":"47ce242c-3874-4d0c-ba59-f71641401374","Type":"ContainerStarted","Data":"0b7e8b57b521227a740b5dcb544034831c8a39c5b28f248d0a39b59c44df42a4"}
May 25 14:01:33.190745 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:33.190701    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 25 14:01:34.294857 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:34.294823    2549 generic.go:358] "Generic (PLEG): container finished" podID="47ce242c-3874-4d0c-ba59-f71641401374" containerID="9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244" exitCode=0
May 25 14:01:34.295334 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:34.294903    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" event={"ID":"47ce242c-3874-4d0c-ba59-f71641401374","Type":"ContainerDied","Data":"9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244"}
May 25 14:01:35.299238 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:35.299194    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" event={"ID":"47ce242c-3874-4d0c-ba59-f71641401374","Type":"ContainerStarted","Data":"a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3"}
May 25 14:01:35.299238 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:35.299250    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" event={"ID":"47ce242c-3874-4d0c-ba59-f71641401374","Type":"ContainerStarted","Data":"253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c"}
May 25 14:01:35.299722 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:35.299569    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:35.299722 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:35.299691    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:35.300861 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:35.300836    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 14:01:35.320667 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:35.320614    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podStartSLOduration=7.32060263 podStartE2EDuration="7.32060263s" podCreationTimestamp="2026-05-25 14:01:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:01:35.319946931 +0000 UTC m=+3445.358704056" watchObservedRunningTime="2026-05-25 14:01:35.32060263 +0000 UTC m=+3445.359359753"
May 25 14:01:36.301950 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:36.301903    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 14:01:38.190804 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:38.190753    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 25 14:01:41.306620 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:41.306589    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:41.307033 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:41.307009    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 14:01:43.190138 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:43.190096    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 25 14:01:43.190555 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:43.190221    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:01:48.190566 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:48.190520    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 25 14:01:51.307845 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:51.307818    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:01:53.190360 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:53.190302    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 25 14:01:58.190025 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:58.189977    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 25 14:01:58.981570 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:58.981543    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:01:59.065186 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.065147    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdk6n\" (UniqueName: \"kubernetes.io/projected/b88b1335-a7a0-4644-b2df-1d8de71a052e-kube-api-access-fdk6n\") pod \"b88b1335-a7a0-4644-b2df-1d8de71a052e\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") "
May 25 14:01:59.065186 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.065202    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls\") pod \"b88b1335-a7a0-4644-b2df-1d8de71a052e\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") "
May 25 14:01:59.065453 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.065224    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b88b1335-a7a0-4644-b2df-1d8de71a052e-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"b88b1335-a7a0-4644-b2df-1d8de71a052e\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") "
May 25 14:01:59.065453 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.065259    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b88b1335-a7a0-4644-b2df-1d8de71a052e-kserve-provision-location\") pod \"b88b1335-a7a0-4644-b2df-1d8de71a052e\" (UID: \"b88b1335-a7a0-4644-b2df-1d8de71a052e\") "
May 25 14:01:59.065674 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.065645    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88b1335-a7a0-4644-b2df-1d8de71a052e-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "b88b1335-a7a0-4644-b2df-1d8de71a052e" (UID: "b88b1335-a7a0-4644-b2df-1d8de71a052e"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:01:59.067486 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.067454    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b88b1335-a7a0-4644-b2df-1d8de71a052e" (UID: "b88b1335-a7a0-4644-b2df-1d8de71a052e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:01:59.067654 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.067629    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88b1335-a7a0-4644-b2df-1d8de71a052e-kube-api-access-fdk6n" (OuterVolumeSpecName: "kube-api-access-fdk6n") pod "b88b1335-a7a0-4644-b2df-1d8de71a052e" (UID: "b88b1335-a7a0-4644-b2df-1d8de71a052e"). InnerVolumeSpecName "kube-api-access-fdk6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:01:59.081990 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.081945    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88b1335-a7a0-4644-b2df-1d8de71a052e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b88b1335-a7a0-4644-b2df-1d8de71a052e" (UID: "b88b1335-a7a0-4644-b2df-1d8de71a052e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:01:59.166193 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.166154    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdk6n\" (UniqueName: \"kubernetes.io/projected/b88b1335-a7a0-4644-b2df-1d8de71a052e-kube-api-access-fdk6n\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:01:59.166193 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.166191    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b88b1335-a7a0-4644-b2df-1d8de71a052e-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:01:59.166401 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.166206    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b88b1335-a7a0-4644-b2df-1d8de71a052e-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:01:59.166401 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.166219    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b88b1335-a7a0-4644-b2df-1d8de71a052e-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:01:59.374545 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.374504    2549 generic.go:358] "Generic (PLEG): container finished" podID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerID="de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3" exitCode=137
May 25 14:01:59.375005 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.374576    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" event={"ID":"b88b1335-a7a0-4644-b2df-1d8de71a052e","Type":"ContainerDied","Data":"de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3"}
May 25 14:01:59.375005 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.374585    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"
May 25 14:01:59.375005 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.374611    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz" event={"ID":"b88b1335-a7a0-4644-b2df-1d8de71a052e","Type":"ContainerDied","Data":"d5695ca10774d3e3a92c77e5f7438ad2aabbffc60d3ee5a6c2a0e13575ea1935"}
May 25 14:01:59.375005 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.374633    2549 scope.go:117] "RemoveContainer" containerID="cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2"
May 25 14:01:59.382761 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.382711    2549 scope.go:117] "RemoveContainer" containerID="de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3"
May 25 14:01:59.390111 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.390089    2549 scope.go:117] "RemoveContainer" containerID="fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e"
May 25 14:01:59.396236 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.396211    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"]
May 25 14:01:59.397956 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.397931    2549 scope.go:117] "RemoveContainer" containerID="cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2"
May 25 14:01:59.398342 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:01:59.398253    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2\": container with ID starting with cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2 not found: ID does not exist" containerID="cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2"
May 25 14:01:59.398510 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.398362    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2"} err="failed to get container status \"cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2\": rpc error: code = NotFound desc = could not find container \"cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2\": container with ID starting with cdfe3b66c1121b82ee14a61c7aacbea51b1b63aea4fb1aa1084e1195a2e847e2 not found: ID does not exist"
May 25 14:01:59.398510 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.398390    2549 scope.go:117] "RemoveContainer" containerID="de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3"
May 25 14:01:59.398997 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:01:59.398973    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3\": container with ID starting with de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3 not found: ID does not exist" containerID="de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3"
May 25 14:01:59.399104 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.399006    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3"} err="failed to get container status \"de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3\": rpc error: code = NotFound desc = could not find container \"de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3\": container with ID starting with de3588f415073cfd02c8b5834a401b1969afce10859b6e0c27cfbc79e0f70cf3 not found: ID does not exist"
May 25 14:01:59.399104 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.399029    2549 scope.go:117] "RemoveContainer" containerID="fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e"
May 25 14:01:59.399352 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:01:59.399332    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e\": container with ID starting with fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e not found: ID does not exist" containerID="fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e"
May 25 14:01:59.399422 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.399366    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e"} err="failed to get container status \"fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e\": rpc error: code = NotFound desc = could not find container \"fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e\": container with ID starting with fdf3851b1c3189b56fb5ee6aef09243bbb17599e0da5bfb3dc4cf37911da216e not found: ID does not exist"
May 25 14:01:59.400093 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:01:59.400070    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-tw8xz"]
May 25 14:02:00.496074 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:00.496029    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" path="/var/lib/kubelet/pods/b88b1335-a7a0-4644-b2df-1d8de71a052e/volumes"
May 25 14:02:09.760938 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.760906    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"]
May 25 14:02:09.761566 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.761282    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kserve-container" containerID="cri-o://253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c" gracePeriod=30
May 25 14:02:09.761566 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.761295    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy" containerID="cri-o://a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3" gracePeriod=30
May 25 14:02:09.835030 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.834997    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"]
May 25 14:02:09.835333 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.835318    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="storage-initializer"
May 25 14:02:09.835333 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.835334    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="storage-initializer"
May 25 14:02:09.835424 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.835344    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy"
May 25 14:02:09.835424 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.835349    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy"
May 25 14:02:09.835424 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.835358    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kserve-container"
May 25 14:02:09.835424 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.835364    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kserve-container"
May 25 14:02:09.835424 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.835409    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kube-rbac-proxy"
May 25 14:02:09.835424 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.835419    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="b88b1335-a7a0-4644-b2df-1d8de71a052e" containerName="kserve-container"
May 25 14:02:09.838237 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.838221    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:09.840351 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.840330    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
May 25 14:02:09.840476 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.840446    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
May 25 14:02:09.849997 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.849974    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"]
May 25 14:02:09.948958 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.948922    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:09.949095 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.948964    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:09.949095 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.948985    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tll7s\" (UniqueName: \"kubernetes.io/projected/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kube-api-access-tll7s\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:09.949095 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:09.949066    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.050582 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.050493    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.050582 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.050533    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tll7s\" (UniqueName: \"kubernetes.io/projected/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kube-api-access-tll7s\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.050770 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.050674    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.050834 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.050790    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.050907 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.050854    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.051376 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.051338    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.053232 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.053213    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.059155 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.059130    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tll7s\" (UniqueName: \"kubernetes.io/projected/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kube-api-access-tll7s\") pod \"isvc-triton-predictor-84bb65d94b-2mfc4\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.148033 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.147989    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:02:10.273170 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.273145    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"]
May 25 14:02:10.275611 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:02:10.275584    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd76e4_f60f_4194_a1de_b1c2acd7bb75.slice/crio-06acba3952863ad5cc6a335bf0063383486e9f50fd6ba430c37cd41bc0cef4a6 WatchSource:0}: Error finding container 06acba3952863ad5cc6a335bf0063383486e9f50fd6ba430c37cd41bc0cef4a6: Status 404 returned error can't find the container with id 06acba3952863ad5cc6a335bf0063383486e9f50fd6ba430c37cd41bc0cef4a6
May 25 14:02:10.417309 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.417251    2549 generic.go:358] "Generic (PLEG): container finished" podID="47ce242c-3874-4d0c-ba59-f71641401374" containerID="a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3" exitCode=2
May 25 14:02:10.417309 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.417297    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" event={"ID":"47ce242c-3874-4d0c-ba59-f71641401374","Type":"ContainerDied","Data":"a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3"}
May 25 14:02:10.418627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.418594    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" event={"ID":"cacd76e4-f60f-4194-a1de-b1c2acd7bb75","Type":"ContainerStarted","Data":"39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296"}
May 25 14:02:10.418627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:10.418620    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" event={"ID":"cacd76e4-f60f-4194-a1de-b1c2acd7bb75","Type":"ContainerStarted","Data":"06acba3952863ad5cc6a335bf0063383486e9f50fd6ba430c37cd41bc0cef4a6"}
May 25 14:02:11.302621 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:11.302580    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 25 14:02:14.430737 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:14.430654    2549 generic.go:358] "Generic (PLEG): container finished" podID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerID="39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296" exitCode=0
May 25 14:02:14.430737 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:14.430718    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" event={"ID":"cacd76e4-f60f-4194-a1de-b1c2acd7bb75","Type":"ContainerDied","Data":"39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296"}
May 25 14:02:16.302945 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:16.302465    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 25 14:02:21.302628 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:21.302495    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 25 14:02:21.303099 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:21.302660    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:02:26.303094 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:26.303044    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 25 14:02:31.302530 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:31.302446    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 25 14:02:36.302983 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:36.302930    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 25 14:02:39.785125 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:02:39.785076    2549 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd76e4_f60f_4194_a1de_b1c2acd7bb75.slice/crio-conmon-39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd76e4_f60f_4194_a1de_b1c2acd7bb75.slice/crio-39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296.scope\": RecentStats: unable to find data in memory cache]"
May 25 14:02:39.785684 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:02:39.785197    2549 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd76e4_f60f_4194_a1de_b1c2acd7bb75.slice/crio-conmon-39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd76e4_f60f_4194_a1de_b1c2acd7bb75.slice/crio-39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296.scope\": RecentStats: unable to find data in memory cache]"
May 25 14:02:40.440919 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.440891    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:02:40.514039 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.514000    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwztj\" (UniqueName: \"kubernetes.io/projected/47ce242c-3874-4d0c-ba59-f71641401374-kube-api-access-xwztj\") pod \"47ce242c-3874-4d0c-ba59-f71641401374\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") "
May 25 14:02:40.514219 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.514108    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ce242c-3874-4d0c-ba59-f71641401374-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"47ce242c-3874-4d0c-ba59-f71641401374\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") "
May 25 14:02:40.514219 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.514143    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ce242c-3874-4d0c-ba59-f71641401374-proxy-tls\") pod \"47ce242c-3874-4d0c-ba59-f71641401374\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") "
May 25 14:02:40.514219 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.514171    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47ce242c-3874-4d0c-ba59-f71641401374-kserve-provision-location\") pod \"47ce242c-3874-4d0c-ba59-f71641401374\" (UID: \"47ce242c-3874-4d0c-ba59-f71641401374\") "
May 25 14:02:40.514696 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.514647    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ce242c-3874-4d0c-ba59-f71641401374-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "47ce242c-3874-4d0c-ba59-f71641401374" (UID: "47ce242c-3874-4d0c-ba59-f71641401374"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:02:40.517518 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.517485    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ce242c-3874-4d0c-ba59-f71641401374-kube-api-access-xwztj" (OuterVolumeSpecName: "kube-api-access-xwztj") pod "47ce242c-3874-4d0c-ba59-f71641401374" (UID: "47ce242c-3874-4d0c-ba59-f71641401374"). InnerVolumeSpecName "kube-api-access-xwztj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:02:40.517644 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.517550    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ce242c-3874-4d0c-ba59-f71641401374-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47ce242c-3874-4d0c-ba59-f71641401374" (UID: "47ce242c-3874-4d0c-ba59-f71641401374"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:02:40.519396 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.519368    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ce242c-3874-4d0c-ba59-f71641401374-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47ce242c-3874-4d0c-ba59-f71641401374" (UID: "47ce242c-3874-4d0c-ba59-f71641401374"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:02:40.538292 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.538222    2549 generic.go:358] "Generic (PLEG): container finished" podID="47ce242c-3874-4d0c-ba59-f71641401374" containerID="253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c" exitCode=137
May 25 14:02:40.538430 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.538356    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"
May 25 14:02:40.538430 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.538372    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" event={"ID":"47ce242c-3874-4d0c-ba59-f71641401374","Type":"ContainerDied","Data":"253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c"}
May 25 14:02:40.538430 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.538405    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n" event={"ID":"47ce242c-3874-4d0c-ba59-f71641401374","Type":"ContainerDied","Data":"0b7e8b57b521227a740b5dcb544034831c8a39c5b28f248d0a39b59c44df42a4"}
May 25 14:02:40.538430 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.538426    2549 scope.go:117] "RemoveContainer" containerID="a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3"
May 25 14:02:40.549175 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.549133    2549 scope.go:117] "RemoveContainer" containerID="253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c"
May 25 14:02:40.559238 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.559214    2549 scope.go:117] "RemoveContainer" containerID="9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244"
May 25 14:02:40.564326 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.564213    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"]
May 25 14:02:40.568789 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.568745    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-pzh5n"]
May 25 14:02:40.569446 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.569374    2549 scope.go:117] "RemoveContainer" containerID="a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3"
May 25 14:02:40.569737 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:02:40.569713    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3\": container with ID starting with a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3 not found: ID does not exist" containerID="a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3"
May 25 14:02:40.569832 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.569745    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3"} err="failed to get container status \"a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3\": rpc error: code = NotFound desc = could not find container \"a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3\": container with ID starting with a84f27ea3514f9482693912f9d84e7150ca3d9331742c89aee198419b37fdcf3 not found: ID does not exist"
May 25 14:02:40.569832 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.569769    2549 scope.go:117] "RemoveContainer" containerID="253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c"
May 25 14:02:40.570221 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:02:40.570195    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c\": container with ID starting with 253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c not found: ID does not exist" containerID="253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c"
May 25 14:02:40.570327 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.570230    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c"} err="failed to get container status \"253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c\": rpc error: code = NotFound desc = could not find container \"253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c\": container with ID starting with 253627d714c9e468347c7aec67902bafc8947e0def92865b7fb153ee02754b5c not found: ID does not exist"
May 25 14:02:40.570327 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.570251    2549 scope.go:117] "RemoveContainer" containerID="9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244"
May 25 14:02:40.570585 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:02:40.570560    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244\": container with ID starting with 9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244 not found: ID does not exist" containerID="9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244"
May 25 14:02:40.570679 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.570592    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244"} err="failed to get container status \"9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244\": rpc error: code = NotFound desc = could not find container \"9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244\": container with ID starting with 9c0b477c67d8a20d1381ef7f5d3e93f4364bf390246d854d85749b8dfb30b244 not found: ID does not exist"
May 25 14:02:40.615100 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.615069    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ce242c-3874-4d0c-ba59-f71641401374-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:02:40.615297 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.615102    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ce242c-3874-4d0c-ba59-f71641401374-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:02:40.615297 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.615122    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47ce242c-3874-4d0c-ba59-f71641401374-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:02:40.615297 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:40.615136    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwztj\" (UniqueName: \"kubernetes.io/projected/47ce242c-3874-4d0c-ba59-f71641401374-kube-api-access-xwztj\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:02:42.497163 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:02:42.497120    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ce242c-3874-4d0c-ba59-f71641401374" path="/var/lib/kubelet/pods/47ce242c-3874-4d0c-ba59-f71641401374/volumes"
May 25 14:04:09.193422 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:09.193401    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 14:04:09.806480 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:09.806445    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" event={"ID":"cacd76e4-f60f-4194-a1de-b1c2acd7bb75","Type":"ContainerStarted","Data":"92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63"}
May 25 14:04:09.806480 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:09.806485    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" event={"ID":"cacd76e4-f60f-4194-a1de-b1c2acd7bb75","Type":"ContainerStarted","Data":"017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7"}
May 25 14:04:09.806702 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:09.806621    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:04:09.825994 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:09.825948    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" podStartSLOduration=6.191498992 podStartE2EDuration="2m0.825933189s" podCreationTimestamp="2026-05-25 14:02:09 +0000 UTC" firstStartedPulling="2026-05-25 14:02:14.431853767 +0000 UTC m=+3484.470610872" lastFinishedPulling="2026-05-25 14:04:09.066287952 +0000 UTC m=+3599.105045069" observedRunningTime="2026-05-25 14:04:09.824799315 +0000 UTC m=+3599.863556442" watchObservedRunningTime="2026-05-25 14:04:09.825933189 +0000 UTC m=+3599.864690312"
May 25 14:04:10.809116 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:10.809075    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:04:10.810295 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:10.810254    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 14:04:11.220413 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:11.220388    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 14:04:11.224602 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:11.224583    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 14:04:11.811993 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:11.811953    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 14:04:16.816181 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:16.816137    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:04:16.816960 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:16.816940    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:04:22.244130 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.244095    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"]
May 25 14:04:22.244682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.244420    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kserve-container" containerID="cri-o://017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7" gracePeriod=30
May 25 14:04:22.244682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.244500    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kube-rbac-proxy" containerID="cri-o://92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63" gracePeriod=30
May 25 14:04:22.358683 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.358651    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"]
May 25 14:04:22.358956 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.358944    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy"
May 25 14:04:22.358999 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.358958    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy"
May 25 14:04:22.358999 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.358966    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kserve-container"
May 25 14:04:22.358999 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.358972    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kserve-container"
May 25 14:04:22.358999 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.358980    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="storage-initializer"
May 25 14:04:22.358999 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.358987    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="storage-initializer"
May 25 14:04:22.359188 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.359035    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kube-rbac-proxy"
May 25 14:04:22.359188 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.359043    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="47ce242c-3874-4d0c-ba59-f71641401374" containerName="kserve-container"
May 25 14:04:22.383440 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.383408    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"]
May 25 14:04:22.383697 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.383514    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.385740 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.385719    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
May 25 14:04:22.385822 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.385744    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
May 25 14:04:22.556458 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.556371    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b2f6ee-72f5-43bf-9033-ff832b085307-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.556458 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.556413    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b2f6ee-72f5-43bf-9033-ff832b085307-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.556458 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.556447    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b2f6ee-72f5-43bf-9033-ff832b085307-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.556676 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.556482    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxrb\" (UniqueName: \"kubernetes.io/projected/a9b2f6ee-72f5-43bf-9033-ff832b085307-kube-api-access-rfxrb\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.657720 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.657685    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b2f6ee-72f5-43bf-9033-ff832b085307-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.657879 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.657729    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b2f6ee-72f5-43bf-9033-ff832b085307-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.657879 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.657753    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b2f6ee-72f5-43bf-9033-ff832b085307-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.657879 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.657776    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxrb\" (UniqueName: \"kubernetes.io/projected/a9b2f6ee-72f5-43bf-9033-ff832b085307-kube-api-access-rfxrb\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.658165 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.658139    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b2f6ee-72f5-43bf-9033-ff832b085307-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.658448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.658431    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b2f6ee-72f5-43bf-9033-ff832b085307-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.660354 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.660337    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b2f6ee-72f5-43bf-9033-ff832b085307-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.667334 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.667308    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxrb\" (UniqueName: \"kubernetes.io/projected/a9b2f6ee-72f5-43bf-9033-ff832b085307-kube-api-access-rfxrb\") pod \"isvc-xgboost-predictor-8689c4cfcc-n4jjs\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.693176 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.693145    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:22.842041 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.842013    2549 generic.go:358] "Generic (PLEG): container finished" podID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerID="92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63" exitCode=2
May 25 14:04:22.842169 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.842112    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" event={"ID":"cacd76e4-f60f-4194-a1de-b1c2acd7bb75","Type":"ContainerDied","Data":"92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63"}
May 25 14:04:22.917517 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:22.917388    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"]
May 25 14:04:23.847311 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:23.847255    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" event={"ID":"a9b2f6ee-72f5-43bf-9033-ff832b085307","Type":"ContainerStarted","Data":"e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f"}
May 25 14:04:23.847311 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:23.847317    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" event={"ID":"a9b2f6ee-72f5-43bf-9033-ff832b085307","Type":"ContainerStarted","Data":"36d76c843bf797d4e3d0092da0a6516f27c3ccbf125d1ca037fab5a89c5c0f89"}
May 25 14:04:25.075504 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.075483    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:04:25.178484 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.178443    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kserve-provision-location\") pod \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") "
May 25 14:04:25.178656 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.178507    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tll7s\" (UniqueName: \"kubernetes.io/projected/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kube-api-access-tll7s\") pod \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") "
May 25 14:04:25.178656 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.178562    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-proxy-tls\") pod \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") "
May 25 14:04:25.178656 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.178618    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-isvc-triton-kube-rbac-proxy-sar-config\") pod \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\" (UID: \"cacd76e4-f60f-4194-a1de-b1c2acd7bb75\") "
May 25 14:04:25.178887 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.178848    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cacd76e4-f60f-4194-a1de-b1c2acd7bb75" (UID: "cacd76e4-f60f-4194-a1de-b1c2acd7bb75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:04:25.179036 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.179013    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "cacd76e4-f60f-4194-a1de-b1c2acd7bb75" (UID: "cacd76e4-f60f-4194-a1de-b1c2acd7bb75"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:04:25.180723 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.180701    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kube-api-access-tll7s" (OuterVolumeSpecName: "kube-api-access-tll7s") pod "cacd76e4-f60f-4194-a1de-b1c2acd7bb75" (UID: "cacd76e4-f60f-4194-a1de-b1c2acd7bb75"). InnerVolumeSpecName "kube-api-access-tll7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:04:25.180806 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.180742    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cacd76e4-f60f-4194-a1de-b1c2acd7bb75" (UID: "cacd76e4-f60f-4194-a1de-b1c2acd7bb75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:04:25.279376 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.279340    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:04:25.279376 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.279372    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:04:25.279566 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.279385    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:04:25.279566 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.279398    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tll7s\" (UniqueName: \"kubernetes.io/projected/cacd76e4-f60f-4194-a1de-b1c2acd7bb75-kube-api-access-tll7s\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:04:25.857519 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.857478    2549 generic.go:358] "Generic (PLEG): container finished" podID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerID="017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7" exitCode=0
May 25 14:04:25.857694 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.857521    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" event={"ID":"cacd76e4-f60f-4194-a1de-b1c2acd7bb75","Type":"ContainerDied","Data":"017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7"}
May 25 14:04:25.857694 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.857560    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"
May 25 14:04:25.857694 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.857574    2549 scope.go:117] "RemoveContainer" containerID="92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63"
May 25 14:04:25.857694 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.857564    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4" event={"ID":"cacd76e4-f60f-4194-a1de-b1c2acd7bb75","Type":"ContainerDied","Data":"06acba3952863ad5cc6a335bf0063383486e9f50fd6ba430c37cd41bc0cef4a6"}
May 25 14:04:25.867333 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.867311    2549 scope.go:117] "RemoveContainer" containerID="017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7"
May 25 14:04:25.875193 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:04:25.875167    2549 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd76e4_f60f_4194_a1de_b1c2acd7bb75.slice/crio-06acba3952863ad5cc6a335bf0063383486e9f50fd6ba430c37cd41bc0cef4a6\": RecentStats: unable to find data in memory cache]"
May 25 14:04:25.875604 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.875500    2549 scope.go:117] "RemoveContainer" containerID="39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296"
May 25 14:04:25.879484 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.879457    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"]
May 25 14:04:25.882582 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.882562    2549 scope.go:117] "RemoveContainer" containerID="92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63"
May 25 14:04:25.882829 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:04:25.882810    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63\": container with ID starting with 92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63 not found: ID does not exist" containerID="92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63"
May 25 14:04:25.882881 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.882837    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63"} err="failed to get container status \"92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63\": rpc error: code = NotFound desc = could not find container \"92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63\": container with ID starting with 92c93b0ed5bee30f5c214ae8c968d721e05b4e43e798a35a0c65c7f138954b63 not found: ID does not exist"
May 25 14:04:25.882881 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.882855    2549 scope.go:117] "RemoveContainer" containerID="017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7"
May 25 14:04:25.883092 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:04:25.883074    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7\": container with ID starting with 017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7 not found: ID does not exist" containerID="017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7"
May 25 14:04:25.883137 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.883099    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7"} err="failed to get container status \"017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7\": rpc error: code = NotFound desc = could not find container \"017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7\": container with ID starting with 017633fe8131c06936caa68cae192f82d8154b350cd99ba26029f1d4f29ed2f7 not found: ID does not exist"
May 25 14:04:25.883137 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.883114    2549 scope.go:117] "RemoveContainer" containerID="39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296"
May 25 14:04:25.883384 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:04:25.883367    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296\": container with ID starting with 39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296 not found: ID does not exist" containerID="39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296"
May 25 14:04:25.883448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.883387    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296"} err="failed to get container status \"39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296\": rpc error: code = NotFound desc = could not find container \"39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296\": container with ID starting with 39090ed6f452ca6380c19635091f6883f70f53501e679a515ea430851ae6b296 not found: ID does not exist"
May 25 14:04:25.884901 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:25.884879    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-2mfc4"]
May 25 14:04:26.496534 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:26.496505    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" path="/var/lib/kubelet/pods/cacd76e4-f60f-4194-a1de-b1c2acd7bb75/volumes"
May 25 14:04:27.865545 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:27.865509    2549 generic.go:358] "Generic (PLEG): container finished" podID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerID="e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f" exitCode=0
May 25 14:04:27.866001 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:27.865564    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" event={"ID":"a9b2f6ee-72f5-43bf-9033-ff832b085307","Type":"ContainerDied","Data":"e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f"}
May 25 14:04:47.928888 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:47.928851    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" event={"ID":"a9b2f6ee-72f5-43bf-9033-ff832b085307","Type":"ContainerStarted","Data":"5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259"}
May 25 14:04:47.928888 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:47.928892    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" event={"ID":"a9b2f6ee-72f5-43bf-9033-ff832b085307","Type":"ContainerStarted","Data":"46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e"}
May 25 14:04:47.929460 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:47.929143    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:47.929460 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:47.929169    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:47.930541 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:47.930514    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:04:47.948156 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:47.948107    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podStartSLOduration=6.650360867 podStartE2EDuration="25.948094786s" podCreationTimestamp="2026-05-25 14:04:22 +0000 UTC" firstStartedPulling="2026-05-25 14:04:27.866735975 +0000 UTC m=+3617.905493077" lastFinishedPulling="2026-05-25 14:04:47.164469877 +0000 UTC m=+3637.203226996" observedRunningTime="2026-05-25 14:04:47.946667383 +0000 UTC m=+3637.985424506" watchObservedRunningTime="2026-05-25 14:04:47.948094786 +0000 UTC m=+3637.986851910"
May 25 14:04:48.931927 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:48.931879    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:04:53.936068 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:53.936037    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:04:53.936613 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:04:53.936584    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:05:03.937343 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:05:03.937264    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:05:13.936817 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:05:13.936770    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:05:23.937191 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:05:23.937100    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:05:33.937453 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:05:33.937414    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:05:43.936955 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:05:43.936919    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:05:53.937265 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:05:53.937227    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:06:02.510887 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.510847    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"]
May 25 14:06:02.511503 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.511158    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" containerID="cri-o://46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e" gracePeriod=30
May 25 14:06:02.511503 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.511203    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kube-rbac-proxy" containerID="cri-o://5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259" gracePeriod=30
May 25 14:06:02.632664 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632620    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"]
May 25 14:06:02.632910 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632897    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="storage-initializer"
May 25 14:06:02.632910 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632911    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="storage-initializer"
May 25 14:06:02.633042 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632919    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kserve-container"
May 25 14:06:02.633042 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632925    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kserve-container"
May 25 14:06:02.633042 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632943    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kube-rbac-proxy"
May 25 14:06:02.633042 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632948    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kube-rbac-proxy"
May 25 14:06:02.633042 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632985    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kube-rbac-proxy"
May 25 14:06:02.633042 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.632996    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="cacd76e4-f60f-4194-a1de-b1c2acd7bb75" containerName="kserve-container"
May 25 14:06:02.635014 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.634992    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.637259 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.637239    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
May 25 14:06:02.637757 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.637738    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 14:06:02.646908 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.646885    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"]
May 25 14:06:02.819053 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.818963    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.819053 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.819001    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkw5j\" (UniqueName: \"kubernetes.io/projected/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kube-api-access-tkw5j\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.819053 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.819023    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.819053 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.819042    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.920449 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.920409    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.920449 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.920449    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tkw5j\" (UniqueName: \"kubernetes.io/projected/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kube-api-access-tkw5j\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.920678 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.920471    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.920678 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.920491    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.920678 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:02.920586    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 14:06:02.920678 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:02.920659    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls podName:05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b nodeName:}" failed. No retries permitted until 2026-05-25 14:06:03.420639915 +0000 UTC m=+3713.459397032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" (UID: "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 14:06:02.920967 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.920948    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.921118 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.921101    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:02.929090 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:02.929071    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkw5j\" (UniqueName: \"kubernetes.io/projected/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kube-api-access-tkw5j\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:03.132162 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:03.132132    2549 generic.go:358] "Generic (PLEG): container finished" podID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerID="5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259" exitCode=2
May 25 14:06:03.132349 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:03.132209    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" event={"ID":"a9b2f6ee-72f5-43bf-9033-ff832b085307","Type":"ContainerDied","Data":"5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259"}
May 25 14:06:03.424078 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:03.423983    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:03.426573 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:03.426551    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:03.546062 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:03.546005    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:03.676351 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:03.676328    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"]
May 25 14:06:03.678886 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:06:03.678859    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a9cfcf_9c0e_49ed_9cf0_0f83ebc2cb1b.slice/crio-ac80387f4a78b110e11919c5bcdd2916bf6be7ab4798146d8247597af49a42a9 WatchSource:0}: Error finding container ac80387f4a78b110e11919c5bcdd2916bf6be7ab4798146d8247597af49a42a9: Status 404 returned error can't find the container with id ac80387f4a78b110e11919c5bcdd2916bf6be7ab4798146d8247597af49a42a9
May 25 14:06:03.932425 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:03.932340    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
May 25 14:06:03.936674 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:03.936650    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 14:06:04.135974 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:04.135935    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" event={"ID":"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b","Type":"ContainerStarted","Data":"9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a"}
May 25 14:06:04.135974 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:04.135972    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" event={"ID":"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b","Type":"ContainerStarted","Data":"ac80387f4a78b110e11919c5bcdd2916bf6be7ab4798146d8247597af49a42a9"}
May 25 14:06:06.055814 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.055789    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:06:06.142918 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.142883    2549 generic.go:358] "Generic (PLEG): container finished" podID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerID="46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e" exitCode=0
May 25 14:06:06.143076 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.142944    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" event={"ID":"a9b2f6ee-72f5-43bf-9033-ff832b085307","Type":"ContainerDied","Data":"46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e"}
May 25 14:06:06.143076 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.142973    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"
May 25 14:06:06.143076 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.142978    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs" event={"ID":"a9b2f6ee-72f5-43bf-9033-ff832b085307","Type":"ContainerDied","Data":"36d76c843bf797d4e3d0092da0a6516f27c3ccbf125d1ca037fab5a89c5c0f89"}
May 25 14:06:06.143076 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.143000    2549 scope.go:117] "RemoveContainer" containerID="5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259"
May 25 14:06:06.150742 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.150721    2549 scope.go:117] "RemoveContainer" containerID="46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e"
May 25 14:06:06.157343 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.157326    2549 scope.go:117] "RemoveContainer" containerID="e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f"
May 25 14:06:06.163659 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.163640    2549 scope.go:117] "RemoveContainer" containerID="5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259"
May 25 14:06:06.163878 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:06.163862    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259\": container with ID starting with 5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259 not found: ID does not exist" containerID="5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259"
May 25 14:06:06.163924 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.163885    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259"} err="failed to get container status \"5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259\": rpc error: code = NotFound desc = could not find container \"5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259\": container with ID starting with 5fcc34c909a2d721bcadf7284caab890f395c53f8ee01a41e1c35cd8ee8f6259 not found: ID does not exist"
May 25 14:06:06.163924 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.163901    2549 scope.go:117] "RemoveContainer" containerID="46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e"
May 25 14:06:06.164133 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:06.164112    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e\": container with ID starting with 46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e not found: ID does not exist" containerID="46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e"
May 25 14:06:06.164197 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.164143    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e"} err="failed to get container status \"46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e\": rpc error: code = NotFound desc = could not find container \"46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e\": container with ID starting with 46b69534de1ac0fe3f05fee2ad92257aa223bede4da1ac9b6085aae20a46b61e not found: ID does not exist"
May 25 14:06:06.164197 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.164170    2549 scope.go:117] "RemoveContainer" containerID="e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f"
May 25 14:06:06.164400 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:06.164384    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f\": container with ID starting with e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f not found: ID does not exist" containerID="e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f"
May 25 14:06:06.164444 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.164404    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f"} err="failed to get container status \"e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f\": rpc error: code = NotFound desc = could not find container \"e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f\": container with ID starting with e721adf45fb7ecdc2a10f79827189e3d4ea2e2ba25a293b5871bf3df2714d88f not found: ID does not exist"
May 25 14:06:06.242806 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.242763    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfxrb\" (UniqueName: \"kubernetes.io/projected/a9b2f6ee-72f5-43bf-9033-ff832b085307-kube-api-access-rfxrb\") pod \"a9b2f6ee-72f5-43bf-9033-ff832b085307\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") "
May 25 14:06:06.242985 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.242821    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b2f6ee-72f5-43bf-9033-ff832b085307-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"a9b2f6ee-72f5-43bf-9033-ff832b085307\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") "
May 25 14:06:06.242985 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.242839    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b2f6ee-72f5-43bf-9033-ff832b085307-proxy-tls\") pod \"a9b2f6ee-72f5-43bf-9033-ff832b085307\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") "
May 25 14:06:06.242985 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.242865    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b2f6ee-72f5-43bf-9033-ff832b085307-kserve-provision-location\") pod \"a9b2f6ee-72f5-43bf-9033-ff832b085307\" (UID: \"a9b2f6ee-72f5-43bf-9033-ff832b085307\") "
May 25 14:06:06.243244 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.243218    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b2f6ee-72f5-43bf-9033-ff832b085307-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "a9b2f6ee-72f5-43bf-9033-ff832b085307" (UID: "a9b2f6ee-72f5-43bf-9033-ff832b085307"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:06:06.243338 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.243321    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b2f6ee-72f5-43bf-9033-ff832b085307-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9b2f6ee-72f5-43bf-9033-ff832b085307" (UID: "a9b2f6ee-72f5-43bf-9033-ff832b085307"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:06:06.245042 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.245006    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b2f6ee-72f5-43bf-9033-ff832b085307-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9b2f6ee-72f5-43bf-9033-ff832b085307" (UID: "a9b2f6ee-72f5-43bf-9033-ff832b085307"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:06:06.245144 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.245081    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b2f6ee-72f5-43bf-9033-ff832b085307-kube-api-access-rfxrb" (OuterVolumeSpecName: "kube-api-access-rfxrb") pod "a9b2f6ee-72f5-43bf-9033-ff832b085307" (UID: "a9b2f6ee-72f5-43bf-9033-ff832b085307"). InnerVolumeSpecName "kube-api-access-rfxrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:06:06.343850 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.343793    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxrb\" (UniqueName: \"kubernetes.io/projected/a9b2f6ee-72f5-43bf-9033-ff832b085307-kube-api-access-rfxrb\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:06:06.343850 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.343844    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b2f6ee-72f5-43bf-9033-ff832b085307-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:06:06.343850 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.343860    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b2f6ee-72f5-43bf-9033-ff832b085307-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:06:06.344107 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.343875    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b2f6ee-72f5-43bf-9033-ff832b085307-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:06:06.464408 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.464373    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"]
May 25 14:06:06.472218 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.470599    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n4jjs"]
May 25 14:06:06.496195 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:06.496160    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" path="/var/lib/kubelet/pods/a9b2f6ee-72f5-43bf-9033-ff832b085307/volumes"
May 25 14:06:08.150952 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:08.150913    2549 generic.go:358] "Generic (PLEG): container finished" podID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerID="9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a" exitCode=0
May 25 14:06:08.151349 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:08.150992    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" event={"ID":"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b","Type":"ContainerDied","Data":"9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a"}
May 25 14:06:09.155702 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:09.155664    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" event={"ID":"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b","Type":"ContainerStarted","Data":"5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676"}
May 25 14:06:09.155702 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:09.155704    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" event={"ID":"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b","Type":"ContainerStarted","Data":"bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25"}
May 25 14:06:09.156112 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:09.155931    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:09.156112 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:09.155982    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:09.176739 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:09.176688    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" podStartSLOduration=7.176676949 podStartE2EDuration="7.176676949s" podCreationTimestamp="2026-05-25 14:06:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:06:09.174690559 +0000 UTC m=+3719.213447684" watchObservedRunningTime="2026-05-25 14:06:09.176676949 +0000 UTC m=+3719.215434073"
May 25 14:06:15.165376 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:15.165343    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:45.169370 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:45.169340    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:52.674947 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.674854    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"]
May 25 14:06:52.675415 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.675317    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kube-rbac-proxy" containerID="cri-o://5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676" gracePeriod=30
May 25 14:06:52.675415 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.675387    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kserve-container" containerID="cri-o://bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25" gracePeriod=30
May 25 14:06:52.804102 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804070    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"]
May 25 14:06:52.804400 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804386    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="storage-initializer"
May 25 14:06:52.804450 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804403    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="storage-initializer"
May 25 14:06:52.804450 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804423    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container"
May 25 14:06:52.804450 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804429    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container"
May 25 14:06:52.804450 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804435    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kube-rbac-proxy"
May 25 14:06:52.804450 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804440    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kube-rbac-proxy"
May 25 14:06:52.804641 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804483    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kube-rbac-proxy"
May 25 14:06:52.804641 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.804496    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9b2f6ee-72f5-43bf-9033-ff832b085307" containerName="kserve-container"
May 25 14:06:52.806566 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.806551    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.809525 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.809498    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 14:06:52.809643 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.809606    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
May 25 14:06:52.823909 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.823887    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"]
May 25 14:06:52.895747 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.895699    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.895747 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.895737    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7km\" (UniqueName: \"kubernetes.io/projected/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kube-api-access-8n7km\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.896016 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.895795    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.896016 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.895813    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e068e219-03b7-4e2f-99b9-fa88fa7b830f-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.997091 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.996983    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.997091 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.997021    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e068e219-03b7-4e2f-99b9-fa88fa7b830f-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.997091 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.997071    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.997091 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.997090    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7km\" (UniqueName: \"kubernetes.io/projected/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kube-api-access-8n7km\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.997498 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:52.997141    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 14:06:52.997498 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:52.997226    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls podName:e068e219-03b7-4e2f-99b9-fa88fa7b830f nodeName:}" failed. No retries permitted until 2026-05-25 14:06:53.497203802 +0000 UTC m=+3763.535960905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls") pod "xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" (UID: "e068e219-03b7-4e2f-99b9-fa88fa7b830f") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 14:06:52.997613 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.997533    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:52.997750 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:52.997730    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e068e219-03b7-4e2f-99b9-fa88fa7b830f-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:53.007289 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:53.007253    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7km\" (UniqueName: \"kubernetes.io/projected/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kube-api-access-8n7km\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:53.271960 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:53.271876    2549 generic.go:358] "Generic (PLEG): container finished" podID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerID="5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676" exitCode=2
May 25 14:06:53.271960 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:53.271947    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" event={"ID":"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b","Type":"ContainerDied","Data":"5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676"}
May 25 14:06:53.500933 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:53.500891    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:53.503344 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:53.503325    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-x7fqj\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:53.716524 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:53.716493    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:53.876458 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:53.876423    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"]
May 25 14:06:53.879810 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:06:53.879779    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode068e219_03b7_4e2f_99b9_fa88fa7b830f.slice/crio-61e4c91ec7785a2f7c67949c41c31311aff6a4fe196fe364a1ea845557e24873 WatchSource:0}: Error finding container 61e4c91ec7785a2f7c67949c41c31311aff6a4fe196fe364a1ea845557e24873: Status 404 returned error can't find the container with id 61e4c91ec7785a2f7c67949c41c31311aff6a4fe196fe364a1ea845557e24873
May 25 14:06:54.275976 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:54.275926    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" event={"ID":"e068e219-03b7-4e2f-99b9-fa88fa7b830f","Type":"ContainerStarted","Data":"b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823"}
May 25 14:06:54.275976 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:54.275974    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" event={"ID":"e068e219-03b7-4e2f-99b9-fa88fa7b830f","Type":"ContainerStarted","Data":"61e4c91ec7785a2f7c67949c41c31311aff6a4fe196fe364a1ea845557e24873"}
May 25 14:06:55.160314 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:55.160249    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
May 25 14:06:58.287914 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:58.287881    2549 generic.go:358] "Generic (PLEG): container finished" podID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerID="b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823" exitCode=0
May 25 14:06:58.288317 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:58.287957    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" event={"ID":"e068e219-03b7-4e2f-99b9-fa88fa7b830f","Type":"ContainerDied","Data":"b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823"}
May 25 14:06:59.211175 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.211153    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:59.293708 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.293610    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" event={"ID":"e068e219-03b7-4e2f-99b9-fa88fa7b830f","Type":"ContainerStarted","Data":"52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2"}
May 25 14:06:59.293708 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.293658    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" event={"ID":"e068e219-03b7-4e2f-99b9-fa88fa7b830f","Type":"ContainerStarted","Data":"01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d"}
May 25 14:06:59.294634 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.293858    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:06:59.295470 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.295445    2549 generic.go:358] "Generic (PLEG): container finished" podID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerID="bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25" exitCode=0
May 25 14:06:59.295585 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.295505    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" event={"ID":"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b","Type":"ContainerDied","Data":"bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25"}
May 25 14:06:59.295585 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.295528    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"
May 25 14:06:59.295585 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.295537    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv" event={"ID":"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b","Type":"ContainerDied","Data":"ac80387f4a78b110e11919c5bcdd2916bf6be7ab4798146d8247597af49a42a9"}
May 25 14:06:59.295585 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.295552    2549 scope.go:117] "RemoveContainer" containerID="5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676"
May 25 14:06:59.303198 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.303178    2549 scope.go:117] "RemoveContainer" containerID="bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25"
May 25 14:06:59.309863 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.309848    2549 scope.go:117] "RemoveContainer" containerID="9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a"
May 25 14:06:59.313154 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.313116    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" podStartSLOduration=7.313104617 podStartE2EDuration="7.313104617s" podCreationTimestamp="2026-05-25 14:06:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:06:59.312469472 +0000 UTC m=+3769.351226608" watchObservedRunningTime="2026-05-25 14:06:59.313104617 +0000 UTC m=+3769.351861741"
May 25 14:06:59.317215 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.317195    2549 scope.go:117] "RemoveContainer" containerID="5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676"
May 25 14:06:59.317487 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:59.317471    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676\": container with ID starting with 5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676 not found: ID does not exist" containerID="5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676"
May 25 14:06:59.317544 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.317496    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676"} err="failed to get container status \"5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676\": rpc error: code = NotFound desc = could not find container \"5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676\": container with ID starting with 5d7a9ea90ec1b3d50e2d362f32dd3c48f688fd51000bbbf1d524f69c7a571676 not found: ID does not exist"
May 25 14:06:59.317544 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.317510    2549 scope.go:117] "RemoveContainer" containerID="bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25"
May 25 14:06:59.317732 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:59.317714    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25\": container with ID starting with bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25 not found: ID does not exist" containerID="bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25"
May 25 14:06:59.317775 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.317740    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25"} err="failed to get container status \"bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25\": rpc error: code = NotFound desc = could not find container \"bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25\": container with ID starting with bd8c8096890ee657fc22ea9bab1b7cf8604adfbe43110af268e2b0ace2855a25 not found: ID does not exist"
May 25 14:06:59.317775 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.317766    2549 scope.go:117] "RemoveContainer" containerID="9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a"
May 25 14:06:59.318018 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:06:59.318000    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a\": container with ID starting with 9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a not found: ID does not exist" containerID="9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a"
May 25 14:06:59.318060 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.318022    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a"} err="failed to get container status \"9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a\": rpc error: code = NotFound desc = could not find container \"9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a\": container with ID starting with 9973378e12d8f24f1c07fbb6284b7311ec0b60b7e421061b7bf8a3757f40512a not found: ID does not exist"
May 25 14:06:59.352011 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.351987    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") "
May 25 14:06:59.352105 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.352035    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kserve-provision-location\") pod \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") "
May 25 14:06:59.352105 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.352080    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls\") pod \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") "
May 25 14:06:59.352198 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.352117    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkw5j\" (UniqueName: \"kubernetes.io/projected/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kube-api-access-tkw5j\") pod \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\" (UID: \"05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b\") "
May 25 14:06:59.352343 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.352322    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" (UID: "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:06:59.352343 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.352329    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" (UID: "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:06:59.352488 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.352385    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:06:59.354223 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.354198    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kube-api-access-tkw5j" (OuterVolumeSpecName: "kube-api-access-tkw5j") pod "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" (UID: "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b"). InnerVolumeSpecName "kube-api-access-tkw5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:06:59.354352 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.354235    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" (UID: "05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:06:59.453236 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.453205    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:06:59.453236 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.453231    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:06:59.453236 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.453243    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkw5j\" (UniqueName: \"kubernetes.io/projected/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b-kube-api-access-tkw5j\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:06:59.618712 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.618680    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"]
May 25 14:06:59.624044 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:06:59.624018    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zrspv"]
May 25 14:07:00.299915 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:00.299884    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:07:00.496195 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:00.496152    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" path="/var/lib/kubelet/pods/05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b/volumes"
May 25 14:07:06.310330 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:06.310304    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:07:36.313999 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:36.313972    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:07:42.877037 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.876996    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"]
May 25 14:07:42.877497 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.877314    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kserve-container" containerID="cri-o://01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d" gracePeriod=30
May 25 14:07:42.877497 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.877380    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kube-rbac-proxy" containerID="cri-o://52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2" gracePeriod=30
May 25 14:07:42.960051 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960014    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"]
May 25 14:07:42.960306 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960293    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="storage-initializer"
May 25 14:07:42.960357 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960308    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="storage-initializer"
May 25 14:07:42.960357 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960323    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kserve-container"
May 25 14:07:42.960357 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960328    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kserve-container"
May 25 14:07:42.960357 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960341    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kube-rbac-proxy"
May 25 14:07:42.960357 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960347    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kube-rbac-proxy"
May 25 14:07:42.960513 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960387    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kube-rbac-proxy"
May 25 14:07:42.960513 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.960398    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="05a9cfcf-9c0e-49ed-9cf0-0f83ebc2cb1b" containerName="kserve-container"
May 25 14:07:42.962868 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.962852    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:42.965013 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.964995    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
May 25 14:07:42.965254 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.965238    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
May 25 14:07:42.974672 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:42.974649    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"]
May 25 14:07:43.075615 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.075568    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.075615 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.075618    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.075840 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.075642    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrmcz\" (UniqueName: \"kubernetes.io/projected/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kube-api-access-wrmcz\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.075840 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.075710    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.176078 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.175991    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.176078 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.176048    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.176078 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.176070    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.176369 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.176095    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wrmcz\" (UniqueName: \"kubernetes.io/projected/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kube-api-access-wrmcz\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.176432 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.176401    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.176773 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.176749    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.178649 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.178630    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.184373 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.184353    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrmcz\" (UniqueName: \"kubernetes.io/projected/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kube-api-access-wrmcz\") pod \"isvc-xgboost-runtime-predictor-779db84d9-grmsj\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.272435 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.272388    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:43.400381 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.400351    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"]
May 25 14:07:43.403514 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:07:43.403488    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffed2be7_d15b_4b42_b09c_b38e78de0e6d.slice/crio-d6c260a4c26587bc44819426729b62e7472b3fd6c76fcc4f7cd77a864c7e4f53 WatchSource:0}: Error finding container d6c260a4c26587bc44819426729b62e7472b3fd6c76fcc4f7cd77a864c7e4f53: Status 404 returned error can't find the container with id d6c260a4c26587bc44819426729b62e7472b3fd6c76fcc4f7cd77a864c7e4f53
May 25 14:07:43.413760 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.413733    2549 generic.go:358] "Generic (PLEG): container finished" podID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerID="52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2" exitCode=2
May 25 14:07:43.413848 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.413803    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" event={"ID":"e068e219-03b7-4e2f-99b9-fa88fa7b830f","Type":"ContainerDied","Data":"52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2"}
May 25 14:07:43.414841 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:43.414818    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" event={"ID":"ffed2be7-d15b-4b42-b09c-b38e78de0e6d","Type":"ContainerStarted","Data":"d6c260a4c26587bc44819426729b62e7472b3fd6c76fcc4f7cd77a864c7e4f53"}
May 25 14:07:44.419318 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:44.419264    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" event={"ID":"ffed2be7-d15b-4b42-b09c-b38e78de0e6d","Type":"ContainerStarted","Data":"c98c23dfa272abe62834175a3849ce80c105707bf2f588964388e818d2dfe5a2"}
May 25 14:07:46.306048 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:46.305995    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
May 25 14:07:46.311448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:46.311416    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.55:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.55:8080: connect: connection refused"
May 25 14:07:47.428192 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:47.428160    2549 generic.go:358] "Generic (PLEG): container finished" podID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerID="c98c23dfa272abe62834175a3849ce80c105707bf2f588964388e818d2dfe5a2" exitCode=0
May 25 14:07:47.428694 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:47.428236    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" event={"ID":"ffed2be7-d15b-4b42-b09c-b38e78de0e6d","Type":"ContainerDied","Data":"c98c23dfa272abe62834175a3849ce80c105707bf2f588964388e818d2dfe5a2"}
May 25 14:07:48.435250 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:48.435208    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" event={"ID":"ffed2be7-d15b-4b42-b09c-b38e78de0e6d","Type":"ContainerStarted","Data":"358936e1004576046557918176fb44464ed6ab69720ecc375e7a8509291f7a3d"}
May 25 14:07:48.435679 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:48.435261    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" event={"ID":"ffed2be7-d15b-4b42-b09c-b38e78de0e6d","Type":"ContainerStarted","Data":"31340355ea0431613e4e46023cda53c81d325698890f349cb90c8255fa712471"}
May 25 14:07:48.435679 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:48.435488    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:48.454171 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:48.454129    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podStartSLOduration=6.454115528 podStartE2EDuration="6.454115528s" podCreationTimestamp="2026-05-25 14:07:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:07:48.453518759 +0000 UTC m=+3818.492275882" watchObservedRunningTime="2026-05-25 14:07:48.454115528 +0000 UTC m=+3818.492872652"
May 25 14:07:49.214088 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.214061    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:07:49.322336 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.322207    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kserve-provision-location\") pod \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") "
May 25 14:07:49.322336 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.322296    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e068e219-03b7-4e2f-99b9-fa88fa7b830f-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") "
May 25 14:07:49.322336 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.322326    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls\") pod \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") "
May 25 14:07:49.322633 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.322350    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n7km\" (UniqueName: \"kubernetes.io/projected/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kube-api-access-8n7km\") pod \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\" (UID: \"e068e219-03b7-4e2f-99b9-fa88fa7b830f\") "
May 25 14:07:49.322692 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.322629    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e068e219-03b7-4e2f-99b9-fa88fa7b830f" (UID: "e068e219-03b7-4e2f-99b9-fa88fa7b830f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:07:49.322692 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.322634    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e068e219-03b7-4e2f-99b9-fa88fa7b830f-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "e068e219-03b7-4e2f-99b9-fa88fa7b830f" (UID: "e068e219-03b7-4e2f-99b9-fa88fa7b830f"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:07:49.324512 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.324481    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kube-api-access-8n7km" (OuterVolumeSpecName: "kube-api-access-8n7km") pod "e068e219-03b7-4e2f-99b9-fa88fa7b830f" (UID: "e068e219-03b7-4e2f-99b9-fa88fa7b830f"). InnerVolumeSpecName "kube-api-access-8n7km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:07:49.324610 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.324552    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e068e219-03b7-4e2f-99b9-fa88fa7b830f" (UID: "e068e219-03b7-4e2f-99b9-fa88fa7b830f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:07:49.422778 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.422738    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n7km\" (UniqueName: \"kubernetes.io/projected/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kube-api-access-8n7km\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:07:49.422778 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.422776    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e068e219-03b7-4e2f-99b9-fa88fa7b830f-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:07:49.423008 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.422791    2549 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e068e219-03b7-4e2f-99b9-fa88fa7b830f-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:07:49.423008 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.422809    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e068e219-03b7-4e2f-99b9-fa88fa7b830f-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:07:49.440201 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.440160    2549 generic.go:358] "Generic (PLEG): container finished" podID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerID="01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d" exitCode=0
May 25 14:07:49.440674 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.440208    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" event={"ID":"e068e219-03b7-4e2f-99b9-fa88fa7b830f","Type":"ContainerDied","Data":"01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d"}
May 25 14:07:49.440674 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.440240    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj" event={"ID":"e068e219-03b7-4e2f-99b9-fa88fa7b830f","Type":"ContainerDied","Data":"61e4c91ec7785a2f7c67949c41c31311aff6a4fe196fe364a1ea845557e24873"}
May 25 14:07:49.440674 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.440250    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"
May 25 14:07:49.440674 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.440257    2549 scope.go:117] "RemoveContainer" containerID="52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2"
May 25 14:07:49.440909 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.440890    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:49.442499 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.442471    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 14:07:49.450681 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.450663    2549 scope.go:117] "RemoveContainer" containerID="01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d"
May 25 14:07:49.457825 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.457808    2549 scope.go:117] "RemoveContainer" containerID="b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823"
May 25 14:07:49.462198 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.462178    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"]
May 25 14:07:49.464921 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.464897    2549 scope.go:117] "RemoveContainer" containerID="52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2"
May 25 14:07:49.465336 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:07:49.465301    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2\": container with ID starting with 52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2 not found: ID does not exist" containerID="52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2"
May 25 14:07:49.465423 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.465349    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2"} err="failed to get container status \"52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2\": rpc error: code = NotFound desc = could not find container \"52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2\": container with ID starting with 52a2bbc70c2688e13688df0aa2cb171a2f1d8dc0548e68e65da0a20c0696b3e2 not found: ID does not exist"
May 25 14:07:49.465423 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.465375    2549 scope.go:117] "RemoveContainer" containerID="01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d"
May 25 14:07:49.465656 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:07:49.465638    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d\": container with ID starting with 01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d not found: ID does not exist" containerID="01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d"
May 25 14:07:49.465709 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.465662    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d"} err="failed to get container status \"01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d\": rpc error: code = NotFound desc = could not find container \"01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d\": container with ID starting with 01240003ea74ca9a038c85f12a95ef85d8e8bd0d4f1443bff449ee4ee41c392d not found: ID does not exist"
May 25 14:07:49.465709 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.465679    2549 scope.go:117] "RemoveContainer" containerID="b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823"
May 25 14:07:49.465956 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:07:49.465940    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823\": container with ID starting with b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823 not found: ID does not exist" containerID="b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823"
May 25 14:07:49.466017 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.465958    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823"} err="failed to get container status \"b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823\": rpc error: code = NotFound desc = could not find container \"b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823\": container with ID starting with b5468063b11ec8251c059fb820417c5be1074ac9f8bb0a26780f8d615823f823 not found: ID does not exist"
May 25 14:07:49.466114 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:49.466098    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-x7fqj"]
May 25 14:07:50.443712 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:50.443667    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 14:07:50.496428 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:50.496390    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" path="/var/lib/kubelet/pods/e068e219-03b7-4e2f-99b9-fa88fa7b830f/volumes"
May 25 14:07:55.449584 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:55.449554    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:07:55.450043 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:07:55.450018    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 14:08:05.450830 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:05.450787    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 14:08:15.451008 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:15.450967    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 14:08:25.450133 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:25.450045    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 14:08:35.450518 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:35.450477    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 14:08:45.450849 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:45.450819    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:08:53.031073 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.031033    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"]
May 25 14:08:53.031477 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.031451    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" containerID="cri-o://31340355ea0431613e4e46023cda53c81d325698890f349cb90c8255fa712471" gracePeriod=30
May 25 14:08:53.031554 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.031503    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kube-rbac-proxy" containerID="cri-o://358936e1004576046557918176fb44464ed6ab69720ecc375e7a8509291f7a3d" gracePeriod=30
May 25 14:08:53.129245 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129206    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"]
May 25 14:08:53.129520 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129505    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kube-rbac-proxy"
May 25 14:08:53.129567 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129524    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kube-rbac-proxy"
May 25 14:08:53.129567 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129541    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="storage-initializer"
May 25 14:08:53.129567 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129547    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="storage-initializer"
May 25 14:08:53.129567 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129561    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kserve-container"
May 25 14:08:53.129567 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129567    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kserve-container"
May 25 14:08:53.129722 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129631    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kserve-container"
May 25 14:08:53.129722 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.129642    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="e068e219-03b7-4e2f-99b9-fa88fa7b830f" containerName="kube-rbac-proxy"
May 25 14:08:53.131648 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.131633    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.133691 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.133666    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 14:08:53.133822 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.133724    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
May 25 14:08:53.142003 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.141964    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"]
May 25 14:08:53.195627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.195586    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c28a40f-9513-4a21-9696-f6a9e4653d29-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.195627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.195623    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c28a40f-9513-4a21-9696-f6a9e4653d29-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.195847 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.195652    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwpk\" (UniqueName: \"kubernetes.io/projected/9c28a40f-9513-4a21-9696-f6a9e4653d29-kube-api-access-cmwpk\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.195847 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.195732    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c28a40f-9513-4a21-9696-f6a9e4653d29-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.296588 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.296482    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c28a40f-9513-4a21-9696-f6a9e4653d29-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.296588 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.296539    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c28a40f-9513-4a21-9696-f6a9e4653d29-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.296588 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.296572    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwpk\" (UniqueName: \"kubernetes.io/projected/9c28a40f-9513-4a21-9696-f6a9e4653d29-kube-api-access-cmwpk\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.296844 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.296674    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c28a40f-9513-4a21-9696-f6a9e4653d29-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.297150 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.297120    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c28a40f-9513-4a21-9696-f6a9e4653d29-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.297385 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.297358    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c28a40f-9513-4a21-9696-f6a9e4653d29-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.299138 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.299121    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c28a40f-9513-4a21-9696-f6a9e4653d29-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.305591 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.305571    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwpk\" (UniqueName: \"kubernetes.io/projected/9c28a40f-9513-4a21-9696-f6a9e4653d29-kube-api-access-cmwpk\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.441924 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.441887    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:53.571922 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.571894    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"]
May 25 14:08:53.574299 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:08:53.574244    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c28a40f_9513_4a21_9696_f6a9e4653d29.slice/crio-e75e3f35177e89521ea6547a64dbd3084abde9a7abbedb00631526ba6652cdaa WatchSource:0}: Error finding container e75e3f35177e89521ea6547a64dbd3084abde9a7abbedb00631526ba6652cdaa: Status 404 returned error can't find the container with id e75e3f35177e89521ea6547a64dbd3084abde9a7abbedb00631526ba6652cdaa
May 25 14:08:53.611915 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.611878    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" event={"ID":"9c28a40f-9513-4a21-9696-f6a9e4653d29","Type":"ContainerStarted","Data":"e75e3f35177e89521ea6547a64dbd3084abde9a7abbedb00631526ba6652cdaa"}
May 25 14:08:53.613714 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.613690    2549 generic.go:358] "Generic (PLEG): container finished" podID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerID="358936e1004576046557918176fb44464ed6ab69720ecc375e7a8509291f7a3d" exitCode=2
May 25 14:08:53.613819 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:53.613769    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" event={"ID":"ffed2be7-d15b-4b42-b09c-b38e78de0e6d","Type":"ContainerDied","Data":"358936e1004576046557918176fb44464ed6ab69720ecc375e7a8509291f7a3d"}
May 25 14:08:54.618688 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:54.618651    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" event={"ID":"9c28a40f-9513-4a21-9696-f6a9e4653d29","Type":"ContainerStarted","Data":"5f5cb1f430ed580ad0fe834510552b2f6f5a07e47a63fdc11045c557afd691ae"}
May 25 14:08:55.444175 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:55.444131    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
May 25 14:08:55.450865 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:55.450829    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 14:08:56.626520 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.626489    2549 generic.go:358] "Generic (PLEG): container finished" podID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerID="31340355ea0431613e4e46023cda53c81d325698890f349cb90c8255fa712471" exitCode=0
May 25 14:08:56.626879 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.626554    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" event={"ID":"ffed2be7-d15b-4b42-b09c-b38e78de0e6d","Type":"ContainerDied","Data":"31340355ea0431613e4e46023cda53c81d325698890f349cb90c8255fa712471"}
May 25 14:08:56.670536 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.670501    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:08:56.723469 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.723440    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") "
May 25 14:08:56.723672 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.723516    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrmcz\" (UniqueName: \"kubernetes.io/projected/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kube-api-access-wrmcz\") pod \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") "
May 25 14:08:56.723672 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.723623    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-proxy-tls\") pod \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") "
May 25 14:08:56.723799 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.723674    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kserve-provision-location\") pod \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\" (UID: \"ffed2be7-d15b-4b42-b09c-b38e78de0e6d\") "
May 25 14:08:56.723882 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.723859    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "ffed2be7-d15b-4b42-b09c-b38e78de0e6d" (UID: "ffed2be7-d15b-4b42-b09c-b38e78de0e6d"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:08:56.724019 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.723990    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffed2be7-d15b-4b42-b09c-b38e78de0e6d" (UID: "ffed2be7-d15b-4b42-b09c-b38e78de0e6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:08:56.725766 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.725737    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffed2be7-d15b-4b42-b09c-b38e78de0e6d" (UID: "ffed2be7-d15b-4b42-b09c-b38e78de0e6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:08:56.725849 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.725793    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kube-api-access-wrmcz" (OuterVolumeSpecName: "kube-api-access-wrmcz") pod "ffed2be7-d15b-4b42-b09c-b38e78de0e6d" (UID: "ffed2be7-d15b-4b42-b09c-b38e78de0e6d"). InnerVolumeSpecName "kube-api-access-wrmcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:08:56.824506 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.824414    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:08:56.824506 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.824456    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrmcz\" (UniqueName: \"kubernetes.io/projected/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kube-api-access-wrmcz\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:08:56.824506 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.824465    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:08:56.824506 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:56.824474    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffed2be7-d15b-4b42-b09c-b38e78de0e6d-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:08:57.631061 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.631030    2549 generic.go:358] "Generic (PLEG): container finished" podID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerID="5f5cb1f430ed580ad0fe834510552b2f6f5a07e47a63fdc11045c557afd691ae" exitCode=0
May 25 14:08:57.631464 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.631099    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" event={"ID":"9c28a40f-9513-4a21-9696-f6a9e4653d29","Type":"ContainerDied","Data":"5f5cb1f430ed580ad0fe834510552b2f6f5a07e47a63fdc11045c557afd691ae"}
May 25 14:08:57.632928 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.632905    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj" event={"ID":"ffed2be7-d15b-4b42-b09c-b38e78de0e6d","Type":"ContainerDied","Data":"d6c260a4c26587bc44819426729b62e7472b3fd6c76fcc4f7cd77a864c7e4f53"}
May 25 14:08:57.632998 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.632945    2549 scope.go:117] "RemoveContainer" containerID="358936e1004576046557918176fb44464ed6ab69720ecc375e7a8509291f7a3d"
May 25 14:08:57.632998 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.632967    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"
May 25 14:08:57.642140 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.642124    2549 scope.go:117] "RemoveContainer" containerID="31340355ea0431613e4e46023cda53c81d325698890f349cb90c8255fa712471"
May 25 14:08:57.655489 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.655464    2549 scope.go:117] "RemoveContainer" containerID="c98c23dfa272abe62834175a3849ce80c105707bf2f588964388e818d2dfe5a2"
May 25 14:08:57.663443 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.663421    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"]
May 25 14:08:57.668873 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:57.668851    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-grmsj"]
May 25 14:08:58.496206 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:58.496163    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" path="/var/lib/kubelet/pods/ffed2be7-d15b-4b42-b09c-b38e78de0e6d/volumes"
May 25 14:08:58.637224 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:58.637182    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" event={"ID":"9c28a40f-9513-4a21-9696-f6a9e4653d29","Type":"ContainerStarted","Data":"99c4181d9904ac210d8fdfbe992d17200d3bf5743862446525db9c1316d0834f"}
May 25 14:08:58.637689 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:58.637234    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" event={"ID":"9c28a40f-9513-4a21-9696-f6a9e4653d29","Type":"ContainerStarted","Data":"0a56379b30a98d5e154673afa8b44df6a23cf1efb13bb4e38ea706cf1412fa10"}
May 25 14:08:58.637689 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:58.637457    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:08:58.658104 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:58.658060    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" podStartSLOduration=5.65804553 podStartE2EDuration="5.65804553s" podCreationTimestamp="2026-05-25 14:08:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:08:58.656494319 +0000 UTC m=+3888.695251444" watchObservedRunningTime="2026-05-25 14:08:58.65804553 +0000 UTC m=+3888.696802653"
May 25 14:08:59.641327 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:08:59.641298    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:09:05.650161 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:05.650131    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:09:11.238231 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:11.238203    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 14:09:11.244060 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:11.244041    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 14:09:35.690685 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:35.690633    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 14:09:45.653430 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:45.653397    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:09:53.302616 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.302534    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"]
May 25 14:09:53.303046 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.302868    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kserve-container" containerID="cri-o://0a56379b30a98d5e154673afa8b44df6a23cf1efb13bb4e38ea706cf1412fa10" gracePeriod=30
May 25 14:09:53.303046 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.302911    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kube-rbac-proxy" containerID="cri-o://99c4181d9904ac210d8fdfbe992d17200d3bf5743862446525db9c1316d0834f" gracePeriod=30
May 25 14:09:53.336294 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336248    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"]
May 25 14:09:53.336534 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336522    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="storage-initializer"
May 25 14:09:53.336577 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336555    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="storage-initializer"
May 25 14:09:53.336577 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336565    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container"
May 25 14:09:53.336577 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336571    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container"
May 25 14:09:53.336673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336579    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kube-rbac-proxy"
May 25 14:09:53.336673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336585    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kube-rbac-proxy"
May 25 14:09:53.336673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336624    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kserve-container"
May 25 14:09:53.336673 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.336634    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="ffed2be7-d15b-4b42-b09c-b38e78de0e6d" containerName="kube-rbac-proxy"
May 25 14:09:53.338666 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.338651    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.340840 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.340819    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
May 25 14:09:53.340928 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.340866    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
May 25 14:09:53.352250 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.352230    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"]
May 25 14:09:53.448758 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.448727    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.448887 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.448774    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8sfw\" (UniqueName: \"kubernetes.io/projected/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kube-api-access-n8sfw\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.448887 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.448824    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.448887 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.448850    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.550198 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.550165    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.550369 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.550214    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n8sfw\" (UniqueName: \"kubernetes.io/projected/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kube-api-access-n8sfw\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.550369 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.550235    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.550369 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.550300    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.550369 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:09:53.550335    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
May 25 14:09:53.550524 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:09:53.550396    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls podName:21cfa254-a7e8-4f13-a10a-ad6f9f678a85 nodeName:}" failed. No retries permitted until 2026-05-25 14:09:54.050377641 +0000 UTC m=+3944.089134749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls") pod "isvc-xgboost-v2-predictor-6fcdd6977c-6967p" (UID: "21cfa254-a7e8-4f13-a10a-ad6f9f678a85") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
May 25 14:09:53.550684 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.550665    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.550943 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.550926    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.560913 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.560857    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8sfw\" (UniqueName: \"kubernetes.io/projected/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kube-api-access-n8sfw\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:53.785834 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.785797    2549 generic.go:358] "Generic (PLEG): container finished" podID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerID="99c4181d9904ac210d8fdfbe992d17200d3bf5743862446525db9c1316d0834f" exitCode=2
May 25 14:09:53.785975 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:53.785877    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" event={"ID":"9c28a40f-9513-4a21-9696-f6a9e4653d29","Type":"ContainerDied","Data":"99c4181d9904ac210d8fdfbe992d17200d3bf5743862446525db9c1316d0834f"}
May 25 14:09:54.053650 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:54.053615    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:54.056115 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:54.056081    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-6967p\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:54.248146 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:54.248106    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:54.376530 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:54.376497    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"]
May 25 14:09:54.379976 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:09:54.379946    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cfa254_a7e8_4f13_a10a_ad6f9f678a85.slice/crio-a9fce6da2223982001f9d3794989c9dddff01bcb86559c958643cbcbd489a82d WatchSource:0}: Error finding container a9fce6da2223982001f9d3794989c9dddff01bcb86559c958643cbcbd489a82d: Status 404 returned error can't find the container with id a9fce6da2223982001f9d3794989c9dddff01bcb86559c958643cbcbd489a82d
May 25 14:09:54.381787 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:54.381767    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 14:09:54.790179 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:54.790140    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" event={"ID":"21cfa254-a7e8-4f13-a10a-ad6f9f678a85","Type":"ContainerStarted","Data":"7b186bf543dfa608c2bc647d5d5962e1a902f045f580397744508504b87cd944"}
May 25 14:09:54.790179 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:54.790181    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" event={"ID":"21cfa254-a7e8-4f13-a10a-ad6f9f678a85","Type":"ContainerStarted","Data":"a9fce6da2223982001f9d3794989c9dddff01bcb86559c958643cbcbd489a82d"}
May 25 14:09:55.645031 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:55.644986    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
May 25 14:09:56.692407 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:56.692364    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 25 14:09:58.801991 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:58.801957    2549 generic.go:358] "Generic (PLEG): container finished" podID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerID="7b186bf543dfa608c2bc647d5d5962e1a902f045f580397744508504b87cd944" exitCode=0
May 25 14:09:58.802397 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:58.802007    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" event={"ID":"21cfa254-a7e8-4f13-a10a-ad6f9f678a85","Type":"ContainerDied","Data":"7b186bf543dfa608c2bc647d5d5962e1a902f045f580397744508504b87cd944"}
May 25 14:09:59.807445 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:59.807401    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" event={"ID":"21cfa254-a7e8-4f13-a10a-ad6f9f678a85","Type":"ContainerStarted","Data":"0da870e5a19ee8e11db842c58760781e7c5d739fbc6e4f6d69e596f771771c48"}
May 25 14:09:59.807989 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:59.807942    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" event={"ID":"21cfa254-a7e8-4f13-a10a-ad6f9f678a85","Type":"ContainerStarted","Data":"a1b1bba40338a8288de71e0292f6f748235cab1697b8bb1f52e1402716c66080"}
May 25 14:09:59.809114 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:59.809089    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:59.809228 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:59.809136    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:09:59.810646 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:59.810619    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:09:59.827307 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:09:59.827251    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podStartSLOduration=6.827239125 podStartE2EDuration="6.827239125s" podCreationTimestamp="2026-05-25 14:09:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:09:59.826203964 +0000 UTC m=+3949.864961128" watchObservedRunningTime="2026-05-25 14:09:59.827239125 +0000 UTC m=+3949.865996250"
May 25 14:10:00.645059 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:00.645011    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
May 25 14:10:00.812664 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:00.812626    2549 generic.go:358] "Generic (PLEG): container finished" podID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerID="0a56379b30a98d5e154673afa8b44df6a23cf1efb13bb4e38ea706cf1412fa10" exitCode=0
May 25 14:10:00.813011 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:00.812693    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" event={"ID":"9c28a40f-9513-4a21-9696-f6a9e4653d29","Type":"ContainerDied","Data":"0a56379b30a98d5e154673afa8b44df6a23cf1efb13bb4e38ea706cf1412fa10"}
May 25 14:10:00.813117 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:00.813095    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:10:00.838656 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:00.838637    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:10:01.007921 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.007838    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c28a40f-9513-4a21-9696-f6a9e4653d29-kserve-provision-location\") pod \"9c28a40f-9513-4a21-9696-f6a9e4653d29\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") "
May 25 14:10:01.007921 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.007894    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c28a40f-9513-4a21-9696-f6a9e4653d29-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"9c28a40f-9513-4a21-9696-f6a9e4653d29\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") "
May 25 14:10:01.008089 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.007930    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmwpk\" (UniqueName: \"kubernetes.io/projected/9c28a40f-9513-4a21-9696-f6a9e4653d29-kube-api-access-cmwpk\") pod \"9c28a40f-9513-4a21-9696-f6a9e4653d29\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") "
May 25 14:10:01.008089 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.007968    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c28a40f-9513-4a21-9696-f6a9e4653d29-proxy-tls\") pod \"9c28a40f-9513-4a21-9696-f6a9e4653d29\" (UID: \"9c28a40f-9513-4a21-9696-f6a9e4653d29\") "
May 25 14:10:01.008250 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.008219    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c28a40f-9513-4a21-9696-f6a9e4653d29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c28a40f-9513-4a21-9696-f6a9e4653d29" (UID: "9c28a40f-9513-4a21-9696-f6a9e4653d29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:10:01.008319 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.008249    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c28a40f-9513-4a21-9696-f6a9e4653d29-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "9c28a40f-9513-4a21-9696-f6a9e4653d29" (UID: "9c28a40f-9513-4a21-9696-f6a9e4653d29"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:10:01.010132 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.010110    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c28a40f-9513-4a21-9696-f6a9e4653d29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c28a40f-9513-4a21-9696-f6a9e4653d29" (UID: "9c28a40f-9513-4a21-9696-f6a9e4653d29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:10:01.010250 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.010229    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c28a40f-9513-4a21-9696-f6a9e4653d29-kube-api-access-cmwpk" (OuterVolumeSpecName: "kube-api-access-cmwpk") pod "9c28a40f-9513-4a21-9696-f6a9e4653d29" (UID: "9c28a40f-9513-4a21-9696-f6a9e4653d29"). InnerVolumeSpecName "kube-api-access-cmwpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:10:01.108824 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.108786    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c28a40f-9513-4a21-9696-f6a9e4653d29-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:10:01.108824 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.108819    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c28a40f-9513-4a21-9696-f6a9e4653d29-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:10:01.108824 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.108832    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmwpk\" (UniqueName: \"kubernetes.io/projected/9c28a40f-9513-4a21-9696-f6a9e4653d29-kube-api-access-cmwpk\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:10:01.109055 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.108843    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c28a40f-9513-4a21-9696-f6a9e4653d29-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:10:01.816941 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.816903    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4" event={"ID":"9c28a40f-9513-4a21-9696-f6a9e4653d29","Type":"ContainerDied","Data":"e75e3f35177e89521ea6547a64dbd3084abde9a7abbedb00631526ba6652cdaa"}
May 25 14:10:01.816941 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.816946    2549 scope.go:117] "RemoveContainer" containerID="99c4181d9904ac210d8fdfbe992d17200d3bf5743862446525db9c1316d0834f"
May 25 14:10:01.817486 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.816962    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"
May 25 14:10:01.824890 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.824872    2549 scope.go:117] "RemoveContainer" containerID="0a56379b30a98d5e154673afa8b44df6a23cf1efb13bb4e38ea706cf1412fa10"
May 25 14:10:01.831954 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.831934    2549 scope.go:117] "RemoveContainer" containerID="5f5cb1f430ed580ad0fe834510552b2f6f5a07e47a63fdc11045c557afd691ae"
May 25 14:10:01.839177 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.839154    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"]
May 25 14:10:01.841629 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:01.841607    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gdpv4"]
May 25 14:10:02.495534 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:02.495491    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" path="/var/lib/kubelet/pods/9c28a40f-9513-4a21-9696-f6a9e4653d29/volumes"
May 25 14:10:05.817505 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:05.817475    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:10:05.817994 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:05.817969    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:10:15.818318 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:15.818254    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:10:25.818890 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:25.818845    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:10:35.818720 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:35.818679    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:10:45.818121 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:45.818074    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:10:55.818477 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:10:55.818438    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:11:05.818461 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:05.818423    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:11:13.431678 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.431628    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"]
May 25 14:11:13.432186 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.432027    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" containerID="cri-o://a1b1bba40338a8288de71e0292f6f748235cab1697b8bb1f52e1402716c66080" gracePeriod=30
May 25 14:11:13.432186 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.432083    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kube-rbac-proxy" containerID="cri-o://0da870e5a19ee8e11db842c58760781e7c5d739fbc6e4f6d69e596f771771c48" gracePeriod=30
May 25 14:11:13.507818 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.507781    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"]
May 25 14:11:13.508081 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.508068    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kserve-container"
May 25 14:11:13.508134 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.508082    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kserve-container"
May 25 14:11:13.508134 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.508095    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="storage-initializer"
May 25 14:11:13.508134 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.508100    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="storage-initializer"
May 25 14:11:13.508134 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.508117    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kube-rbac-proxy"
May 25 14:11:13.508134 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.508122    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kube-rbac-proxy"
May 25 14:11:13.508314 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.508162    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kserve-container"
May 25 14:11:13.508314 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.508170    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c28a40f-9513-4a21-9696-f6a9e4653d29" containerName="kube-rbac-proxy"
May 25 14:11:13.511188 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.511173    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.513562 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.513537    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 25 14:11:13.513710 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.513537    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
May 25 14:11:13.513710 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.513549    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
May 25 14:11:13.521085 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.521054    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"]
May 25 14:11:13.630873 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.630836    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd349478-7dfa-499c-b97c-8be91c328773-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.630873 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.630876    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd349478-7dfa-499c-b97c-8be91c328773-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.631102 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.630989    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd349478-7dfa-499c-b97c-8be91c328773-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.631102 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.631039    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vm84\" (UniqueName: \"kubernetes.io/projected/fd349478-7dfa-499c-b97c-8be91c328773-kube-api-access-7vm84\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.732373 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.732285    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd349478-7dfa-499c-b97c-8be91c328773-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.732373 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.732320    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd349478-7dfa-499c-b97c-8be91c328773-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.732373 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.732364    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd349478-7dfa-499c-b97c-8be91c328773-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.732639 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.732393    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7vm84\" (UniqueName: \"kubernetes.io/projected/fd349478-7dfa-499c-b97c-8be91c328773-kube-api-access-7vm84\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.732762 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.732743    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd349478-7dfa-499c-b97c-8be91c328773-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.733021 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.733003    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd349478-7dfa-499c-b97c-8be91c328773-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.735040 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.735019    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd349478-7dfa-499c-b97c-8be91c328773-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.741578 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.741548    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vm84\" (UniqueName: \"kubernetes.io/projected/fd349478-7dfa-499c-b97c-8be91c328773-kube-api-access-7vm84\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-w22q2\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.821851 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.821819    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:13.944683 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:13.944656    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"]
May 25 14:11:13.947806 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:11:13.947776    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd349478_7dfa_499c_b97c_8be91c328773.slice/crio-2a829be263a9bc95b0affe173a058e8bd810811bf4f4708bcf50f854c7086581 WatchSource:0}: Error finding container 2a829be263a9bc95b0affe173a058e8bd810811bf4f4708bcf50f854c7086581: Status 404 returned error can't find the container with id 2a829be263a9bc95b0affe173a058e8bd810811bf4f4708bcf50f854c7086581
May 25 14:11:14.013780 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:14.013749    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" event={"ID":"fd349478-7dfa-499c-b97c-8be91c328773","Type":"ContainerStarted","Data":"cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5"}
May 25 14:11:14.013889 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:14.013790    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" event={"ID":"fd349478-7dfa-499c-b97c-8be91c328773","Type":"ContainerStarted","Data":"2a829be263a9bc95b0affe173a058e8bd810811bf4f4708bcf50f854c7086581"}
May 25 14:11:14.015640 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:14.015617    2549 generic.go:358] "Generic (PLEG): container finished" podID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerID="0da870e5a19ee8e11db842c58760781e7c5d739fbc6e4f6d69e596f771771c48" exitCode=2
May 25 14:11:14.015719 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:14.015650    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" event={"ID":"21cfa254-a7e8-4f13-a10a-ad6f9f678a85","Type":"ContainerDied","Data":"0da870e5a19ee8e11db842c58760781e7c5d739fbc6e4f6d69e596f771771c48"}
May 25 14:11:15.019378 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:15.019340    2549 generic.go:358] "Generic (PLEG): container finished" podID="fd349478-7dfa-499c-b97c-8be91c328773" containerID="cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5" exitCode=0
May 25 14:11:15.019822 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:15.019426    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" event={"ID":"fd349478-7dfa-499c-b97c-8be91c328773","Type":"ContainerDied","Data":"cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5"}
May 25 14:11:15.813508 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:15.813463    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
May 25 14:11:15.818108 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:15.818081    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 25 14:11:16.024024 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:16.023987    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" event={"ID":"fd349478-7dfa-499c-b97c-8be91c328773","Type":"ContainerStarted","Data":"9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39"}
May 25 14:11:16.024024 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:16.024031    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" event={"ID":"fd349478-7dfa-499c-b97c-8be91c328773","Type":"ContainerStarted","Data":"97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c"}
May 25 14:11:16.024454 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:16.024095    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:16.044935 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:16.044893    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podStartSLOduration=3.044878516 podStartE2EDuration="3.044878516s" podCreationTimestamp="2026-05-25 14:11:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:11:16.044533348 +0000 UTC m=+4026.083290474" watchObservedRunningTime="2026-05-25 14:11:16.044878516 +0000 UTC m=+4026.083635637"
May 25 14:11:17.028814 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.028781    2549 generic.go:358] "Generic (PLEG): container finished" podID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerID="a1b1bba40338a8288de71e0292f6f748235cab1697b8bb1f52e1402716c66080" exitCode=0
May 25 14:11:17.029156 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.028857    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" event={"ID":"21cfa254-a7e8-4f13-a10a-ad6f9f678a85","Type":"ContainerDied","Data":"a1b1bba40338a8288de71e0292f6f748235cab1697b8bb1f52e1402716c66080"}
May 25 14:11:17.029238 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.029224    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:17.030383 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.030360    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 14:11:17.071160 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.071138    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:11:17.160184 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.160091    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8sfw\" (UniqueName: \"kubernetes.io/projected/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kube-api-access-n8sfw\") pod \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") "
May 25 14:11:17.160184 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.160138    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") "
May 25 14:11:17.160184 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.160179    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls\") pod \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") "
May 25 14:11:17.160504 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.160233    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kserve-provision-location\") pod \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\" (UID: \"21cfa254-a7e8-4f13-a10a-ad6f9f678a85\") "
May 25 14:11:17.160599 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.160567    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21cfa254-a7e8-4f13-a10a-ad6f9f678a85" (UID: "21cfa254-a7e8-4f13-a10a-ad6f9f678a85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:11:17.160658 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.160602    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "21cfa254-a7e8-4f13-a10a-ad6f9f678a85" (UID: "21cfa254-a7e8-4f13-a10a-ad6f9f678a85"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:11:17.162434 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.162412    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21cfa254-a7e8-4f13-a10a-ad6f9f678a85" (UID: "21cfa254-a7e8-4f13-a10a-ad6f9f678a85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:11:17.162558 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.162537    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kube-api-access-n8sfw" (OuterVolumeSpecName: "kube-api-access-n8sfw") pod "21cfa254-a7e8-4f13-a10a-ad6f9f678a85" (UID: "21cfa254-a7e8-4f13-a10a-ad6f9f678a85"). InnerVolumeSpecName "kube-api-access-n8sfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:11:17.261613 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.261575    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8sfw\" (UniqueName: \"kubernetes.io/projected/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kube-api-access-n8sfw\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:11:17.261613 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.261607    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:11:17.261613 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.261619    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:11:17.261848 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:17.261630    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21cfa254-a7e8-4f13-a10a-ad6f9f678a85-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:11:18.033452 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.033413    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p" event={"ID":"21cfa254-a7e8-4f13-a10a-ad6f9f678a85","Type":"ContainerDied","Data":"a9fce6da2223982001f9d3794989c9dddff01bcb86559c958643cbcbd489a82d"}
May 25 14:11:18.033452 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.033438    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"
May 25 14:11:18.033944 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.033466    2549 scope.go:117] "RemoveContainer" containerID="0da870e5a19ee8e11db842c58760781e7c5d739fbc6e4f6d69e596f771771c48"
May 25 14:11:18.033944 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.033875    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 14:11:18.041149 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.041116    2549 scope.go:117] "RemoveContainer" containerID="a1b1bba40338a8288de71e0292f6f748235cab1697b8bb1f52e1402716c66080"
May 25 14:11:18.048112 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.048094    2549 scope.go:117] "RemoveContainer" containerID="7b186bf543dfa608c2bc647d5d5962e1a902f045f580397744508504b87cd944"
May 25 14:11:18.054056 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.054033    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"]
May 25 14:11:18.059505 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.059486    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-6967p"]
May 25 14:11:18.496321 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:18.496261    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" path="/var/lib/kubelet/pods/21cfa254-a7e8-4f13-a10a-ad6f9f678a85/volumes"
May 25 14:11:23.037951 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:23.037925    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:11:23.038421 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:23.038394    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 14:11:33.038634 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:33.038591    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 14:11:43.039028 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:43.038988    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 14:11:53.038753 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:11:53.038707    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 14:12:03.038355 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:03.038313    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 14:12:13.039377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:13.039339    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 14:12:23.038927 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.038897    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:12:23.632156 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.632117    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"]
May 25 14:12:23.632500 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.632470    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container" containerID="cri-o://97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c" gracePeriod=30
May 25 14:12:23.632614 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.632495    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kube-rbac-proxy" containerID="cri-o://9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39" gracePeriod=30
May 25 14:12:23.745487 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745459    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"]
May 25 14:12:23.745752 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745739    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container"
May 25 14:12:23.745804 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745754    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container"
May 25 14:12:23.745804 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745763    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="storage-initializer"
May 25 14:12:23.745804 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745769    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="storage-initializer"
May 25 14:12:23.745804 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745778    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kube-rbac-proxy"
May 25 14:12:23.745804 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745783    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kube-rbac-proxy"
May 25 14:12:23.745958 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745826    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kserve-container"
May 25 14:12:23.745958 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.745835    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="21cfa254-a7e8-4f13-a10a-ad6f9f678a85" containerName="kube-rbac-proxy"
May 25 14:12:23.747749 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.747732    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.749912 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.749890    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
May 25 14:12:23.750018 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.749967    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 14:12:23.750018 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.750009    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
May 25 14:12:23.759569 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.759541    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"]
May 25 14:12:23.861658 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.861620    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92fd35aa-93d8-40b5-94d6-ff3a9645c969-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.861825 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.861667    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hth\" (UniqueName: \"kubernetes.io/projected/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kube-api-access-w2hth\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.861825 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.861720    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.861825 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.861772    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.861825 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.861808    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.962853 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.962765    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.962853 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.962809    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92fd35aa-93d8-40b5-94d6-ff3a9645c969-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.963027 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.962931    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hth\" (UniqueName: \"kubernetes.io/projected/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kube-api-access-w2hth\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.963027 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.962974    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.963027 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.963013    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.963192 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.963175    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.963650 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.963628    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.963742 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.963663    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.965457 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.965440    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92fd35aa-93d8-40b5-94d6-ff3a9645c969-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:23.971554 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:23.971535    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hth\" (UniqueName: \"kubernetes.io/projected/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kube-api-access-w2hth\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:24.057522 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:24.057478    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:24.187942 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:24.187911    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"]
May 25 14:12:24.190331 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:12:24.190302    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fd35aa_93d8_40b5_94d6_ff3a9645c969.slice/crio-12ca06cdbbcc99df9384b601e8229dff1de82ccf0aa5ca37b79f3793e5c5915c WatchSource:0}: Error finding container 12ca06cdbbcc99df9384b601e8229dff1de82ccf0aa5ca37b79f3793e5c5915c: Status 404 returned error can't find the container with id 12ca06cdbbcc99df9384b601e8229dff1de82ccf0aa5ca37b79f3793e5c5915c
May 25 14:12:24.212318 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:24.212255    2549 generic.go:358] "Generic (PLEG): container finished" podID="fd349478-7dfa-499c-b97c-8be91c328773" containerID="9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39" exitCode=2
May 25 14:12:24.212432 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:24.212315    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" event={"ID":"fd349478-7dfa-499c-b97c-8be91c328773","Type":"ContainerDied","Data":"9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39"}
May 25 14:12:24.213558 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:24.213495    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" event={"ID":"92fd35aa-93d8-40b5-94d6-ff3a9645c969","Type":"ContainerStarted","Data":"12ca06cdbbcc99df9384b601e8229dff1de82ccf0aa5ca37b79f3793e5c5915c"}
May 25 14:12:25.217735 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:25.217699    2549 generic.go:358] "Generic (PLEG): container finished" podID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerID="56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77" exitCode=0
May 25 14:12:25.218157 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:25.217742    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" event={"ID":"92fd35aa-93d8-40b5-94d6-ff3a9645c969","Type":"ContainerDied","Data":"56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77"}
May 25 14:12:26.221935 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:26.221900    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" event={"ID":"92fd35aa-93d8-40b5-94d6-ff3a9645c969","Type":"ContainerStarted","Data":"2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09"}
May 25 14:12:26.221935 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:26.221936    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" event={"ID":"92fd35aa-93d8-40b5-94d6-ff3a9645c969","Type":"ContainerStarted","Data":"c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456"}
May 25 14:12:26.222371 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:26.222048    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:26.241060 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:26.241013    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podStartSLOduration=3.240998363 podStartE2EDuration="3.240998363s" podCreationTimestamp="2026-05-25 14:12:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:12:26.239744815 +0000 UTC m=+4096.278501940" watchObservedRunningTime="2026-05-25 14:12:26.240998363 +0000 UTC m=+4096.279755487"
May 25 14:12:27.229450 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:27.228577    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:27.229920 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:27.229626    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 25 14:12:28.076101 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.076075    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:12:28.196132 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.196037    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd349478-7dfa-499c-b97c-8be91c328773-kserve-provision-location\") pod \"fd349478-7dfa-499c-b97c-8be91c328773\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") "
May 25 14:12:28.196132 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.196074    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd349478-7dfa-499c-b97c-8be91c328773-proxy-tls\") pod \"fd349478-7dfa-499c-b97c-8be91c328773\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") "
May 25 14:12:28.196370 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.196139    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vm84\" (UniqueName: \"kubernetes.io/projected/fd349478-7dfa-499c-b97c-8be91c328773-kube-api-access-7vm84\") pod \"fd349478-7dfa-499c-b97c-8be91c328773\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") "
May 25 14:12:28.196370 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.196161    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd349478-7dfa-499c-b97c-8be91c328773-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"fd349478-7dfa-499c-b97c-8be91c328773\" (UID: \"fd349478-7dfa-499c-b97c-8be91c328773\") "
May 25 14:12:28.196457 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.196373    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd349478-7dfa-499c-b97c-8be91c328773-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd349478-7dfa-499c-b97c-8be91c328773" (UID: "fd349478-7dfa-499c-b97c-8be91c328773"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:12:28.196699 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.196664    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd349478-7dfa-499c-b97c-8be91c328773-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "fd349478-7dfa-499c-b97c-8be91c328773" (UID: "fd349478-7dfa-499c-b97c-8be91c328773"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:12:28.198376 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.198345    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd349478-7dfa-499c-b97c-8be91c328773-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd349478-7dfa-499c-b97c-8be91c328773" (UID: "fd349478-7dfa-499c-b97c-8be91c328773"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:12:28.198471 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.198446    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd349478-7dfa-499c-b97c-8be91c328773-kube-api-access-7vm84" (OuterVolumeSpecName: "kube-api-access-7vm84") pod "fd349478-7dfa-499c-b97c-8be91c328773" (UID: "fd349478-7dfa-499c-b97c-8be91c328773"). InnerVolumeSpecName "kube-api-access-7vm84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:12:28.230910 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.230880    2549 generic.go:358] "Generic (PLEG): container finished" podID="fd349478-7dfa-499c-b97c-8be91c328773" containerID="97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c" exitCode=0
May 25 14:12:28.231289 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.230958    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"
May 25 14:12:28.231289 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.230961    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" event={"ID":"fd349478-7dfa-499c-b97c-8be91c328773","Type":"ContainerDied","Data":"97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c"}
May 25 14:12:28.231289 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.231001    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" event={"ID":"fd349478-7dfa-499c-b97c-8be91c328773","Type":"ContainerDied","Data":"2a829be263a9bc95b0affe173a058e8bd810811bf4f4708bcf50f854c7086581"}
May 25 14:12:28.231289 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.231017    2549 scope.go:117] "RemoveContainer" containerID="9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39"
May 25 14:12:28.231511 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.231485    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 25 14:12:28.238826 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.238694    2549 scope.go:117] "RemoveContainer" containerID="97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c"
May 25 14:12:28.245994 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.245978    2549 scope.go:117] "RemoveContainer" containerID="cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5"
May 25 14:12:28.252011 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.251988    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"]
May 25 14:12:28.252940 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.252921    2549 scope.go:117] "RemoveContainer" containerID="9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39"
May 25 14:12:28.253220 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:12:28.253194    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39\": container with ID starting with 9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39 not found: ID does not exist" containerID="9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39"
May 25 14:12:28.253315 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.253234    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39"} err="failed to get container status \"9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39\": rpc error: code = NotFound desc = could not find container \"9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39\": container with ID starting with 9500a3405945a45f000fd9b9188918e389175a6472ba2f4999002984c0b11f39 not found: ID does not exist"
May 25 14:12:28.253315 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.253258    2549 scope.go:117] "RemoveContainer" containerID="97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c"
May 25 14:12:28.253836 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:12:28.253819    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c\": container with ID starting with 97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c not found: ID does not exist" containerID="97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c"
May 25 14:12:28.253900 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.253841    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c"} err="failed to get container status \"97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c\": rpc error: code = NotFound desc = could not find container \"97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c\": container with ID starting with 97e1f5de565864b391007b889a466f682845ba18b0b356d3cf842b6fabaf454c not found: ID does not exist"
May 25 14:12:28.253900 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.253857    2549 scope.go:117] "RemoveContainer" containerID="cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5"
May 25 14:12:28.254129 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:12:28.254112    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5\": container with ID starting with cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5 not found: ID does not exist" containerID="cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5"
May 25 14:12:28.254190 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.254133    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5"} err="failed to get container status \"cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5\": rpc error: code = NotFound desc = could not find container \"cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5\": container with ID starting with cc5e2897c24a5ae3f08d9a939f478495433ebf6c49dc17586dfef8c0266a41e5 not found: ID does not exist"
May 25 14:12:28.254228 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.254193    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2"]
May 25 14:12:28.297748 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.297725    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vm84\" (UniqueName: \"kubernetes.io/projected/fd349478-7dfa-499c-b97c-8be91c328773-kube-api-access-7vm84\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:12:28.297882 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.297754    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd349478-7dfa-499c-b97c-8be91c328773-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:12:28.297882 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.297771    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd349478-7dfa-499c-b97c-8be91c328773-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:12:28.297882 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.297787    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd349478-7dfa-499c-b97c-8be91c328773-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:12:28.495569 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:28.495497    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd349478-7dfa-499c-b97c-8be91c328773" path="/var/lib/kubelet/pods/fd349478-7dfa-499c-b97c-8be91c328773/volumes"
May 25 14:12:29.034647 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:29.034604    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-w22q2" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": context deadline exceeded"
May 25 14:12:33.235694 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:33.235664    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:12:33.236138 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:33.236110    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 25 14:12:43.236510 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:43.236469    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 25 14:12:53.236386 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:12:53.236264    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 25 14:13:03.236129 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:03.236086    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 25 14:13:13.236604 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:13.236561    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 25 14:13:23.238073 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:23.238028    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 25 14:13:33.237338 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:33.237306    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:13:33.825395 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:33.825360    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"]
May 25 14:13:33.825687 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:33.825637    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container" containerID="cri-o://c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456" gracePeriod=30
May 25 14:13:33.825803 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:33.825693    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kube-rbac-proxy" containerID="cri-o://2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09" gracePeriod=30
May 25 14:13:34.416189 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.416156    2549 generic.go:358] "Generic (PLEG): container finished" podID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerID="2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09" exitCode=2
May 25 14:13:34.416665 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.416236    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" event={"ID":"92fd35aa-93d8-40b5-94d6-ff3a9645c969","Type":"ContainerDied","Data":"2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09"}
May 25 14:13:34.908689 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.908655    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"]
May 25 14:13:34.908924 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.908912    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container"
May 25 14:13:34.908981 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.908926    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container"
May 25 14:13:34.908981 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.908937    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="storage-initializer"
May 25 14:13:34.908981 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.908942    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="storage-initializer"
May 25 14:13:34.908981 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.908948    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kube-rbac-proxy"
May 25 14:13:34.908981 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.908954    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kube-rbac-proxy"
May 25 14:13:34.909135 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.909004    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kube-rbac-proxy"
May 25 14:13:34.909135 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.909013    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="fd349478-7dfa-499c-b97c-8be91c328773" containerName="kserve-container"
May 25 14:13:34.911028 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.911011    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:34.914100 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.914078    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
May 25 14:13:34.914611 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.914592    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
May 25 14:13:34.923637 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.923617    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"]
May 25 14:13:34.984393 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.984363    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk4hv\" (UniqueName: \"kubernetes.io/projected/a0a26ab0-56f7-4e89-afbf-532e961654ab-kube-api-access-pk4hv\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:34.984550 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.984409    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a26ab0-56f7-4e89-afbf-532e961654ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:34.984550 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.984433    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a26ab0-56f7-4e89-afbf-532e961654ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:34.984550 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:34.984528    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a26ab0-56f7-4e89-afbf-532e961654ab-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.085193 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.085165    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a26ab0-56f7-4e89-afbf-532e961654ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.085398 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.085213    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a26ab0-56f7-4e89-afbf-532e961654ab-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.085398 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.085244    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pk4hv\" (UniqueName: \"kubernetes.io/projected/a0a26ab0-56f7-4e89-afbf-532e961654ab-kube-api-access-pk4hv\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.085398 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.085306    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a26ab0-56f7-4e89-afbf-532e961654ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.085730 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.085711    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a26ab0-56f7-4e89-afbf-532e961654ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.085995 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.085972    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a26ab0-56f7-4e89-afbf-532e961654ab-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.087743 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.087726    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a26ab0-56f7-4e89-afbf-532e961654ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.097896 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.097875    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk4hv\" (UniqueName: \"kubernetes.io/projected/a0a26ab0-56f7-4e89-afbf-532e961654ab-kube-api-access-pk4hv\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.221151 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.221056    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:35.352636 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.352605    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"]
May 25 14:13:35.354445 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:13:35.354412    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a26ab0_56f7_4e89_afbf_532e961654ab.slice/crio-a1ad77e6a3026544049bee78fe9280b27eddb601843aeae57aa90572044f868f WatchSource:0}: Error finding container a1ad77e6a3026544049bee78fe9280b27eddb601843aeae57aa90572044f868f: Status 404 returned error can't find the container with id a1ad77e6a3026544049bee78fe9280b27eddb601843aeae57aa90572044f868f
May 25 14:13:35.421030 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.420997    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789" event={"ID":"a0a26ab0-56f7-4e89-afbf-532e961654ab","Type":"ContainerStarted","Data":"6b0abaccba77b8b9553e1a70258b3732f806e15967f10d7264757cd741df3a6e"}
May 25 14:13:35.421030 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:35.421035    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789" event={"ID":"a0a26ab0-56f7-4e89-afbf-532e961654ab","Type":"ContainerStarted","Data":"a1ad77e6a3026544049bee78fe9280b27eddb601843aeae57aa90572044f868f"}
May 25 14:13:38.066330 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.066308    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:13:38.106363 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.106336    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") "
May 25 14:13:38.106529 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.106374    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92fd35aa-93d8-40b5-94d6-ff3a9645c969-proxy-tls\") pod \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") "
May 25 14:13:38.106529 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.106399    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2hth\" (UniqueName: \"kubernetes.io/projected/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kube-api-access-w2hth\") pod \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") "
May 25 14:13:38.106529 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.106463    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kserve-provision-location\") pod \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") "
May 25 14:13:38.106529 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.106496    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-cabundle-cert\") pod \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\" (UID: \"92fd35aa-93d8-40b5-94d6-ff3a9645c969\") "
May 25 14:13:38.106837 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.106738    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "92fd35aa-93d8-40b5-94d6-ff3a9645c969" (UID: "92fd35aa-93d8-40b5-94d6-ff3a9645c969"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:13:38.106960 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.106809    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92fd35aa-93d8-40b5-94d6-ff3a9645c969" (UID: "92fd35aa-93d8-40b5-94d6-ff3a9645c969"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:13:38.106960 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.106868    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92fd35aa-93d8-40b5-94d6-ff3a9645c969" (UID: "92fd35aa-93d8-40b5-94d6-ff3a9645c969"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:13:38.108659 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.108641    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fd35aa-93d8-40b5-94d6-ff3a9645c969-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92fd35aa-93d8-40b5-94d6-ff3a9645c969" (UID: "92fd35aa-93d8-40b5-94d6-ff3a9645c969"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:13:38.108659 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.108642    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kube-api-access-w2hth" (OuterVolumeSpecName: "kube-api-access-w2hth") pod "92fd35aa-93d8-40b5-94d6-ff3a9645c969" (UID: "92fd35aa-93d8-40b5-94d6-ff3a9645c969"). InnerVolumeSpecName "kube-api-access-w2hth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:13:38.207448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.207367    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:38.207448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.207397    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92fd35aa-93d8-40b5-94d6-ff3a9645c969-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:38.207448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.207411    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2hth\" (UniqueName: \"kubernetes.io/projected/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kube-api-access-w2hth\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:38.207448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.207420    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92fd35aa-93d8-40b5-94d6-ff3a9645c969-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:38.207448 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.207428    2549 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92fd35aa-93d8-40b5-94d6-ff3a9645c969-cabundle-cert\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:38.430962 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.430924    2549 generic.go:358] "Generic (PLEG): container finished" podID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerID="c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456" exitCode=0
May 25 14:13:38.431161 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.430975    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" event={"ID":"92fd35aa-93d8-40b5-94d6-ff3a9645c969","Type":"ContainerDied","Data":"c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456"}
May 25 14:13:38.431161 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.431020    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"
May 25 14:13:38.431161 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.431033    2549 scope.go:117] "RemoveContainer" containerID="2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09"
May 25 14:13:38.431161 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.431023    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc" event={"ID":"92fd35aa-93d8-40b5-94d6-ff3a9645c969","Type":"ContainerDied","Data":"12ca06cdbbcc99df9384b601e8229dff1de82ccf0aa5ca37b79f3793e5c5915c"}
May 25 14:13:38.440309 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.440291    2549 scope.go:117] "RemoveContainer" containerID="c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456"
May 25 14:13:38.447115 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.447095    2549 scope.go:117] "RemoveContainer" containerID="56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77"
May 25 14:13:38.453285 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.453242    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"]
May 25 14:13:38.456555 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.456241    2549 scope.go:117] "RemoveContainer" containerID="2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09"
May 25 14:13:38.456722 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:13:38.456695    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09\": container with ID starting with 2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09 not found: ID does not exist" containerID="2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09"
May 25 14:13:38.456805 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.456738    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09"} err="failed to get container status \"2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09\": rpc error: code = NotFound desc = could not find container \"2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09\": container with ID starting with 2fa2d8d3a9fda2e8bb38f8c8234b44e46a8222a9787536eeb4f897c71853fb09 not found: ID does not exist"
May 25 14:13:38.456805 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.456764    2549 scope.go:117] "RemoveContainer" containerID="c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456"
May 25 14:13:38.457081 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:13:38.457054    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456\": container with ID starting with c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456 not found: ID does not exist" containerID="c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456"
May 25 14:13:38.457163 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.457102    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456"} err="failed to get container status \"c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456\": rpc error: code = NotFound desc = could not find container \"c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456\": container with ID starting with c976ad771720e7cd2bed69da6c5d0f97c6f092cf55c6518af21740c7d7827456 not found: ID does not exist"
May 25 14:13:38.457163 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.457125    2549 scope.go:117] "RemoveContainer" containerID="56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77"
May 25 14:13:38.457351 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.457174    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-sq6dc"]
May 25 14:13:38.457474 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:13:38.457429    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77\": container with ID starting with 56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77 not found: ID does not exist" containerID="56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77"
May 25 14:13:38.457516 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.457463    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77"} err="failed to get container status \"56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77\": rpc error: code = NotFound desc = could not find container \"56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77\": container with ID starting with 56cf7f5027152b22a8d733f99739a6c29fdd62c787069347b653c2722a39da77 not found: ID does not exist"
May 25 14:13:38.495683 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:38.495645    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" path="/var/lib/kubelet/pods/92fd35aa-93d8-40b5-94d6-ff3a9645c969/volumes"
May 25 14:13:42.445957 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:42.445926    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789_a0a26ab0-56f7-4e89-afbf-532e961654ab/storage-initializer/0.log"
May 25 14:13:42.446350 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:42.445965    2549 generic.go:358] "Generic (PLEG): container finished" podID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerID="6b0abaccba77b8b9553e1a70258b3732f806e15967f10d7264757cd741df3a6e" exitCode=1
May 25 14:13:42.446350 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:42.446024    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789" event={"ID":"a0a26ab0-56f7-4e89-afbf-532e961654ab","Type":"ContainerDied","Data":"6b0abaccba77b8b9553e1a70258b3732f806e15967f10d7264757cd741df3a6e"}
May 25 14:13:43.449956 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:43.449930    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789_a0a26ab0-56f7-4e89-afbf-532e961654ab/storage-initializer/0.log"
May 25 14:13:43.450421 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:43.450029    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789" event={"ID":"a0a26ab0-56f7-4e89-afbf-532e961654ab","Type":"ContainerStarted","Data":"69a220df9cd162f67746fb3fb986694377b8bcad045b0ecfb2be2d341a033e64"}
May 25 14:13:44.887530 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:44.887493    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"]
May 25 14:13:44.887962 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:44.887728    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789" podUID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerName="storage-initializer" containerID="cri-o://69a220df9cd162f67746fb3fb986694377b8bcad045b0ecfb2be2d341a033e64" gracePeriod=30
May 25 14:13:45.947563 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947525    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"]
May 25 14:13:45.948007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947807    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="storage-initializer"
May 25 14:13:45.948007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947818    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="storage-initializer"
May 25 14:13:45.948007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947832    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kube-rbac-proxy"
May 25 14:13:45.948007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947837    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kube-rbac-proxy"
May 25 14:13:45.948007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947847    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container"
May 25 14:13:45.948007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947854    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container"
May 25 14:13:45.948007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947894    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kube-rbac-proxy"
May 25 14:13:45.948007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.947902    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="92fd35aa-93d8-40b5-94d6-ff3a9645c969" containerName="kserve-container"
May 25 14:13:45.949937 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.949920    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:45.952410 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.952387    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 14:13:45.952631 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.952614    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
May 25 14:13:45.953084 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.953065    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
May 25 14:13:45.962553 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:45.962531    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"]
May 25 14:13:46.065054 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.065015    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.065054 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.065054    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048c92fe-2834-44a8-80bd-cc381dd539dd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.065315 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.065079    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048c92fe-2834-44a8-80bd-cc381dd539dd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.065315 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.065137    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4fmw\" (UniqueName: \"kubernetes.io/projected/048c92fe-2834-44a8-80bd-cc381dd539dd-kube-api-access-s4fmw\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.065315 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.065160    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.166389 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.166355    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s4fmw\" (UniqueName: \"kubernetes.io/projected/048c92fe-2834-44a8-80bd-cc381dd539dd-kube-api-access-s4fmw\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.166525 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.166402    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.166525 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.166449    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.166525 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.166468    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048c92fe-2834-44a8-80bd-cc381dd539dd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.166525 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.166489    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048c92fe-2834-44a8-80bd-cc381dd539dd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.166984 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.166958    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048c92fe-2834-44a8-80bd-cc381dd539dd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.167155 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.167136    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.167442 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.167198    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.169088 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.169066    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048c92fe-2834-44a8-80bd-cc381dd539dd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.175680 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.175658    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4fmw\" (UniqueName: \"kubernetes.io/projected/048c92fe-2834-44a8-80bd-cc381dd539dd-kube-api-access-s4fmw\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.260095 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.260005    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:46.389852 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.389674    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"]
May 25 14:13:46.392298 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:13:46.392243    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048c92fe_2834_44a8_80bd_cc381dd539dd.slice/crio-aa616b25d71fc191a20de6e3c8e5580b6b1f3fbd49b5e34dadcc16080e75d503 WatchSource:0}: Error finding container aa616b25d71fc191a20de6e3c8e5580b6b1f3fbd49b5e34dadcc16080e75d503: Status 404 returned error can't find the container with id aa616b25d71fc191a20de6e3c8e5580b6b1f3fbd49b5e34dadcc16080e75d503
May 25 14:13:46.459795 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.459763    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" event={"ID":"048c92fe-2834-44a8-80bd-cc381dd539dd","Type":"ContainerStarted","Data":"6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616"}
May 25 14:13:46.459924 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:46.459805    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" event={"ID":"048c92fe-2834-44a8-80bd-cc381dd539dd","Type":"ContainerStarted","Data":"aa616b25d71fc191a20de6e3c8e5580b6b1f3fbd49b5e34dadcc16080e75d503"}
May 25 14:13:47.463501 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:47.463464    2549 generic.go:358] "Generic (PLEG): container finished" podID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerID="6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616" exitCode=0
May 25 14:13:47.463899 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:47.463508    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" event={"ID":"048c92fe-2834-44a8-80bd-cc381dd539dd","Type":"ContainerDied","Data":"6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616"}
May 25 14:13:48.469790 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.469754    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" event={"ID":"048c92fe-2834-44a8-80bd-cc381dd539dd","Type":"ContainerStarted","Data":"8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a"}
May 25 14:13:48.470236 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.469802    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" event={"ID":"048c92fe-2834-44a8-80bd-cc381dd539dd","Type":"ContainerStarted","Data":"f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d"}
May 25 14:13:48.470236 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.469839    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:48.471181 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.471166    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789_a0a26ab0-56f7-4e89-afbf-532e961654ab/storage-initializer/1.log"
May 25 14:13:48.471564 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.471547    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789_a0a26ab0-56f7-4e89-afbf-532e961654ab/storage-initializer/0.log"
May 25 14:13:48.471627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.471581    2549 generic.go:358] "Generic (PLEG): container finished" podID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerID="69a220df9cd162f67746fb3fb986694377b8bcad045b0ecfb2be2d341a033e64" exitCode=1
May 25 14:13:48.471627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.471602    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789" event={"ID":"a0a26ab0-56f7-4e89-afbf-532e961654ab","Type":"ContainerDied","Data":"69a220df9cd162f67746fb3fb986694377b8bcad045b0ecfb2be2d341a033e64"}
May 25 14:13:48.471627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.471623    2549 scope.go:117] "RemoveContainer" containerID="6b0abaccba77b8b9553e1a70258b3732f806e15967f10d7264757cd741df3a6e"
May 25 14:13:48.491440 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.491388    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podStartSLOduration=3.49136882 podStartE2EDuration="3.49136882s" podCreationTimestamp="2026-05-25 14:13:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:13:48.488890713 +0000 UTC m=+4178.527647839" watchObservedRunningTime="2026-05-25 14:13:48.49136882 +0000 UTC m=+4178.530125947"
May 25 14:13:48.523867 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.523843    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789_a0a26ab0-56f7-4e89-afbf-532e961654ab/storage-initializer/1.log"
May 25 14:13:48.524014 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.523906    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:48.582739 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.582701    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a26ab0-56f7-4e89-afbf-532e961654ab-proxy-tls\") pod \"a0a26ab0-56f7-4e89-afbf-532e961654ab\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") "
May 25 14:13:48.582910 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.582767    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a26ab0-56f7-4e89-afbf-532e961654ab-kserve-provision-location\") pod \"a0a26ab0-56f7-4e89-afbf-532e961654ab\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") "
May 25 14:13:48.582910 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.582810    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a26ab0-56f7-4e89-afbf-532e961654ab-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"a0a26ab0-56f7-4e89-afbf-532e961654ab\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") "
May 25 14:13:48.582910 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.582833    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk4hv\" (UniqueName: \"kubernetes.io/projected/a0a26ab0-56f7-4e89-afbf-532e961654ab-kube-api-access-pk4hv\") pod \"a0a26ab0-56f7-4e89-afbf-532e961654ab\" (UID: \"a0a26ab0-56f7-4e89-afbf-532e961654ab\") "
May 25 14:13:48.583096 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.583044    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a26ab0-56f7-4e89-afbf-532e961654ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0a26ab0-56f7-4e89-afbf-532e961654ab" (UID: "a0a26ab0-56f7-4e89-afbf-532e961654ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:13:48.583232 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.583200    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a26ab0-56f7-4e89-afbf-532e961654ab-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "a0a26ab0-56f7-4e89-afbf-532e961654ab" (UID: "a0a26ab0-56f7-4e89-afbf-532e961654ab"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:13:48.584945 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.584925    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a26ab0-56f7-4e89-afbf-532e961654ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0a26ab0-56f7-4e89-afbf-532e961654ab" (UID: "a0a26ab0-56f7-4e89-afbf-532e961654ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:13:48.585095 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.585077    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a26ab0-56f7-4e89-afbf-532e961654ab-kube-api-access-pk4hv" (OuterVolumeSpecName: "kube-api-access-pk4hv") pod "a0a26ab0-56f7-4e89-afbf-532e961654ab" (UID: "a0a26ab0-56f7-4e89-afbf-532e961654ab"). InnerVolumeSpecName "kube-api-access-pk4hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:13:48.683575 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.683483    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a26ab0-56f7-4e89-afbf-532e961654ab-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:48.683575 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.683513    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a26ab0-56f7-4e89-afbf-532e961654ab-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:48.683575 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.683527    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk4hv\" (UniqueName: \"kubernetes.io/projected/a0a26ab0-56f7-4e89-afbf-532e961654ab-kube-api-access-pk4hv\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:48.683575 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:48.683540    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a26ab0-56f7-4e89-afbf-532e961654ab-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:13:49.478261 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:49.478231    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789_a0a26ab0-56f7-4e89-afbf-532e961654ab/storage-initializer/1.log"
May 25 14:13:49.478661 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:49.478365    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"
May 25 14:13:49.478661 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:49.478374    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789" event={"ID":"a0a26ab0-56f7-4e89-afbf-532e961654ab","Type":"ContainerDied","Data":"a1ad77e6a3026544049bee78fe9280b27eddb601843aeae57aa90572044f868f"}
May 25 14:13:49.478661 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:49.478408    2549 scope.go:117] "RemoveContainer" containerID="69a220df9cd162f67746fb3fb986694377b8bcad045b0ecfb2be2d341a033e64"
May 25 14:13:49.478923 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:49.478907    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:49.480285 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:49.480237    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 25 14:13:49.515214 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:49.515180    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"]
May 25 14:13:49.518687 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:49.518662    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-js789"]
May 25 14:13:50.487791 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:50.487749    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 25 14:13:50.496006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:50.495973    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a26ab0-56f7-4e89-afbf-532e961654ab" path="/var/lib/kubelet/pods/a0a26ab0-56f7-4e89-afbf-532e961654ab/volumes"
May 25 14:13:55.492815 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:55.492786    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:13:55.493297 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:13:55.493252    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 25 14:14:05.493401 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:05.493347    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 25 14:14:11.256660 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:11.256627    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 14:14:11.263303 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:11.263280    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 14:14:15.493559 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:15.493519    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 25 14:14:25.493558 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:25.493463    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 25 14:14:35.494130 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:35.494076    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 25 14:14:45.493506 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:45.493465    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 25 14:14:55.494244 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:55.494214    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:14:55.954143 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:55.954101    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"]
May 25 14:14:55.954622 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:55.954572    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container" containerID="cri-o://f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d" gracePeriod=30
May 25 14:14:55.954715 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:55.954654    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kube-rbac-proxy" containerID="cri-o://8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a" gracePeriod=30
May 25 14:14:56.670162 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:56.670121    2549 generic.go:358] "Generic (PLEG): container finished" podID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerID="8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a" exitCode=2
May 25 14:14:56.670581 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:56.670191    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" event={"ID":"048c92fe-2834-44a8-80bd-cc381dd539dd","Type":"ContainerDied","Data":"8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a"}
May 25 14:14:57.085128 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.085055    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"]
May 25 14:14:57.085340 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.085328    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerName="storage-initializer"
May 25 14:14:57.085414 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.085342    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerName="storage-initializer"
May 25 14:14:57.085451 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.085420    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerName="storage-initializer"
May 25 14:14:57.085451 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.085428    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerName="storage-initializer"
May 25 14:14:57.085513 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.085475    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerName="storage-initializer"
May 25 14:14:57.085513 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.085482    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a26ab0-56f7-4e89-afbf-532e961654ab" containerName="storage-initializer"
May 25 14:14:57.088470 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.088453    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.090485 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.090463    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
May 25 14:14:57.090587 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.090472    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
May 25 14:14:57.098783 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.098759    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"]
May 25 14:14:57.190491 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.190454    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167b5203-12a1-4cde-a49a-596de4670058-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.190682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.190512    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hlw\" (UniqueName: \"kubernetes.io/projected/167b5203-12a1-4cde-a49a-596de4670058-kube-api-access-l9hlw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.190682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.190538    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/167b5203-12a1-4cde-a49a-596de4670058-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.190682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.190596    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.290921 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.290883    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.291023 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.290950    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167b5203-12a1-4cde-a49a-596de4670058-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.291023 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.290988    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hlw\" (UniqueName: \"kubernetes.io/projected/167b5203-12a1-4cde-a49a-596de4670058-kube-api-access-l9hlw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.291100 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.291020    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/167b5203-12a1-4cde-a49a-596de4670058-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.291100 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:14:57.291037    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
May 25 14:14:57.291200 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:14:57.291115    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls podName:167b5203-12a1-4cde-a49a-596de4670058 nodeName:}" failed. No retries permitted until 2026-05-25 14:14:57.791093289 +0000 UTC m=+4247.829850403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr" (UID: "167b5203-12a1-4cde-a49a-596de4670058") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
May 25 14:14:57.291444 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.291425    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167b5203-12a1-4cde-a49a-596de4670058-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.291758 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.291739    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/167b5203-12a1-4cde-a49a-596de4670058-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.299189 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.299166    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hlw\" (UniqueName: \"kubernetes.io/projected/167b5203-12a1-4cde-a49a-596de4670058-kube-api-access-l9hlw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.795155 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.795101    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.797583 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.797563    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:57.998819 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:57.998774    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:14:58.121012 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:58.120980    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"]
May 25 14:14:58.124112 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:14:58.124086    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167b5203_12a1_4cde_a49a_596de4670058.slice/crio-000154c758581f58bb86d8d0d4c492174f1988ed5fb2a25870afc02655b091ea WatchSource:0}: Error finding container 000154c758581f58bb86d8d0d4c492174f1988ed5fb2a25870afc02655b091ea: Status 404 returned error can't find the container with id 000154c758581f58bb86d8d0d4c492174f1988ed5fb2a25870afc02655b091ea
May 25 14:14:58.126628 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:58.126613    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 14:14:58.676596 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:58.676562    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr" event={"ID":"167b5203-12a1-4cde-a49a-596de4670058","Type":"ContainerStarted","Data":"47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff"}
May 25 14:14:58.676596 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:14:58.676599    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr" event={"ID":"167b5203-12a1-4cde-a49a-596de4670058","Type":"ContainerStarted","Data":"000154c758581f58bb86d8d0d4c492174f1988ed5fb2a25870afc02655b091ea"}
May 25 14:15:00.403902 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.403874    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:15:00.515571 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.515490    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048c92fe-2834-44a8-80bd-cc381dd539dd-proxy-tls\") pod \"048c92fe-2834-44a8-80bd-cc381dd539dd\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") "
May 25 14:15:00.515571 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.515533    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4fmw\" (UniqueName: \"kubernetes.io/projected/048c92fe-2834-44a8-80bd-cc381dd539dd-kube-api-access-s4fmw\") pod \"048c92fe-2834-44a8-80bd-cc381dd539dd\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") "
May 25 14:15:00.515799 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.515606    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-cabundle-cert\") pod \"048c92fe-2834-44a8-80bd-cc381dd539dd\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") "
May 25 14:15:00.515799 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.515638    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"048c92fe-2834-44a8-80bd-cc381dd539dd\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") "
May 25 14:15:00.515966 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.515820    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048c92fe-2834-44a8-80bd-cc381dd539dd-kserve-provision-location\") pod \"048c92fe-2834-44a8-80bd-cc381dd539dd\" (UID: \"048c92fe-2834-44a8-80bd-cc381dd539dd\") "
May 25 14:15:00.516066 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.515981    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "048c92fe-2834-44a8-80bd-cc381dd539dd" (UID: "048c92fe-2834-44a8-80bd-cc381dd539dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:15:00.516066 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.516043    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "048c92fe-2834-44a8-80bd-cc381dd539dd" (UID: "048c92fe-2834-44a8-80bd-cc381dd539dd"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:15:00.516205 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.516103    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048c92fe-2834-44a8-80bd-cc381dd539dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "048c92fe-2834-44a8-80bd-cc381dd539dd" (UID: "048c92fe-2834-44a8-80bd-cc381dd539dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:15:00.517723 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.517701    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048c92fe-2834-44a8-80bd-cc381dd539dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "048c92fe-2834-44a8-80bd-cc381dd539dd" (UID: "048c92fe-2834-44a8-80bd-cc381dd539dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:15:00.517838 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.517825    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048c92fe-2834-44a8-80bd-cc381dd539dd-kube-api-access-s4fmw" (OuterVolumeSpecName: "kube-api-access-s4fmw") pod "048c92fe-2834-44a8-80bd-cc381dd539dd" (UID: "048c92fe-2834-44a8-80bd-cc381dd539dd"). InnerVolumeSpecName "kube-api-access-s4fmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:15:00.616946 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.616891    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048c92fe-2834-44a8-80bd-cc381dd539dd-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:00.616946 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.616939    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4fmw\" (UniqueName: \"kubernetes.io/projected/048c92fe-2834-44a8-80bd-cc381dd539dd-kube-api-access-s4fmw\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:00.616946 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.616949    2549 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-cabundle-cert\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:00.616946 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.616960    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048c92fe-2834-44a8-80bd-cc381dd539dd-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:00.616946 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.616969    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048c92fe-2834-44a8-80bd-cc381dd539dd-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:00.684100 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.684061    2549 generic.go:358] "Generic (PLEG): container finished" podID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerID="f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d" exitCode=0
May 25 14:15:00.684309 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.684111    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" event={"ID":"048c92fe-2834-44a8-80bd-cc381dd539dd","Type":"ContainerDied","Data":"f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d"}
May 25 14:15:00.684309 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.684133    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8" event={"ID":"048c92fe-2834-44a8-80bd-cc381dd539dd","Type":"ContainerDied","Data":"aa616b25d71fc191a20de6e3c8e5580b6b1f3fbd49b5e34dadcc16080e75d503"}
May 25 14:15:00.684309 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.684137    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"
May 25 14:15:00.684309 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.684147    2549 scope.go:117] "RemoveContainer" containerID="8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a"
May 25 14:15:00.692443 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.692425    2549 scope.go:117] "RemoveContainer" containerID="f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d"
May 25 14:15:00.699489 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.699469    2549 scope.go:117] "RemoveContainer" containerID="6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616"
May 25 14:15:00.705099 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.705078    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"]
May 25 14:15:00.707215 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.707195    2549 scope.go:117] "RemoveContainer" containerID="8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a"
May 25 14:15:00.707489 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:15:00.707469    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a\": container with ID starting with 8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a not found: ID does not exist" containerID="8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a"
May 25 14:15:00.707565 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.707501    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a"} err="failed to get container status \"8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a\": rpc error: code = NotFound desc = could not find container \"8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a\": container with ID starting with 8ecc5eb025d6f56f67bca35b06f3baffd0ac66559bb8a3536b71e773d580d39a not found: ID does not exist"
May 25 14:15:00.707565 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.707524    2549 scope.go:117] "RemoveContainer" containerID="f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d"
May 25 14:15:00.707753 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:15:00.707737    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d\": container with ID starting with f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d not found: ID does not exist" containerID="f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d"
May 25 14:15:00.707792 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.707760    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d"} err="failed to get container status \"f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d\": rpc error: code = NotFound desc = could not find container \"f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d\": container with ID starting with f335c32ed384de93fb3159016b434187fc7d5ef112ca5cc41e709855dea6b73d not found: ID does not exist"
May 25 14:15:00.707792 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.707786    2549 scope.go:117] "RemoveContainer" containerID="6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616"
May 25 14:15:00.707987 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:15:00.707968    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616\": container with ID starting with 6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616 not found: ID does not exist" containerID="6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616"
May 25 14:15:00.708029 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.707993    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616"} err="failed to get container status \"6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616\": rpc error: code = NotFound desc = could not find container \"6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616\": container with ID starting with 6422f22f9b4d577f43985b6500704abf15b40d63f4c41f61b51c7afadfe2f616 not found: ID does not exist"
May 25 14:15:00.716640 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:00.716614    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-5zgb8"]
May 25 14:15:02.495952 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:02.495909    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" path="/var/lib/kubelet/pods/048c92fe-2834-44a8-80bd-cc381dd539dd/volumes"
May 25 14:15:04.696235 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:04.696208    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr_167b5203-12a1-4cde-a49a-596de4670058/storage-initializer/0.log"
May 25 14:15:04.696622 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:04.696244    2549 generic.go:358] "Generic (PLEG): container finished" podID="167b5203-12a1-4cde-a49a-596de4670058" containerID="47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff" exitCode=1
May 25 14:15:04.696622 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:04.696322    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr" event={"ID":"167b5203-12a1-4cde-a49a-596de4670058","Type":"ContainerDied","Data":"47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff"}
May 25 14:15:05.700217 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:05.700186    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr_167b5203-12a1-4cde-a49a-596de4670058/storage-initializer/0.log"
May 25 14:15:05.700705 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:05.700240    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr" event={"ID":"167b5203-12a1-4cde-a49a-596de4670058","Type":"ContainerStarted","Data":"e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7"}
May 25 14:15:07.048418 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:07.048377    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"]
May 25 14:15:07.048901 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:07.048705    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr" podUID="167b5203-12a1-4cde-a49a-596de4670058" containerName="storage-initializer" containerID="cri-o://e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7" gracePeriod=30
May 25 14:15:08.101974 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.101938    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"]
May 25 14:15:08.102377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.102212    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kube-rbac-proxy"
May 25 14:15:08.102377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.102224    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kube-rbac-proxy"
May 25 14:15:08.102377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.102235    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="storage-initializer"
May 25 14:15:08.102377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.102240    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="storage-initializer"
May 25 14:15:08.102377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.102281    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container"
May 25 14:15:08.102377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.102288    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container"
May 25 14:15:08.102377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.102333    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kserve-container"
May 25 14:15:08.102377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.102347    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="048c92fe-2834-44a8-80bd-cc381dd539dd" containerName="kube-rbac-proxy"
May 25 14:15:08.105262 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.105244    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.107365 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.107341    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
May 25 14:15:08.107365 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.107355    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 14:15:08.107564 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.107441    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
May 25 14:15:08.113937 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.113914    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"]
May 25 14:15:08.174465 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.174431    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65804066-9483-4e60-9944-78d068bee94a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.174648 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.174480    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.174648 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.174569    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.174648 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.174611    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.174776 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.174651    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krg8q\" (UniqueName: \"kubernetes.io/projected/65804066-9483-4e60-9944-78d068bee94a-kube-api-access-krg8q\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.275717 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.275685    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65804066-9483-4e60-9944-78d068bee94a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.275857 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.275730    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.275857 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.275755    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.275857 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.275780    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.275857 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.275819    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-krg8q\" (UniqueName: \"kubernetes.io/projected/65804066-9483-4e60-9944-78d068bee94a-kube-api-access-krg8q\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.276044 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:15:08.275949    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
May 25 14:15:08.276044 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:15:08.276012    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls podName:65804066-9483-4e60-9944-78d068bee94a nodeName:}" failed. No retries permitted until 2026-05-25 14:15:08.775991727 +0000 UTC m=+4258.814748828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" (UID: "65804066-9483-4e60-9944-78d068bee94a") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
May 25 14:15:08.276175 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.276149    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65804066-9483-4e60-9944-78d068bee94a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.276495 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.276477    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.276549 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.276533    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.284372 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.284353    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krg8q\" (UniqueName: \"kubernetes.io/projected/65804066-9483-4e60-9944-78d068bee94a-kube-api-access-krg8q\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.780306 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.780249    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:08.782814 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:08.782794    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:09.016056 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.016014    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:09.142086 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.142053    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"]
May 25 14:15:09.145109 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:15:09.145078    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65804066_9483_4e60_9944_78d068bee94a.slice/crio-76f495fe32db74bd793c44abf89037dcac87124f1ec852955cfdac196cc669ee WatchSource:0}: Error finding container 76f495fe32db74bd793c44abf89037dcac87124f1ec852955cfdac196cc669ee: Status 404 returned error can't find the container with id 76f495fe32db74bd793c44abf89037dcac87124f1ec852955cfdac196cc669ee
May 25 14:15:09.689770 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.689747    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr_167b5203-12a1-4cde-a49a-596de4670058/storage-initializer/1.log"
May 25 14:15:09.690145 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.690128    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr_167b5203-12a1-4cde-a49a-596de4670058/storage-initializer/0.log"
May 25 14:15:09.690245 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.690194    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:15:09.711468 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.711435    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" event={"ID":"65804066-9483-4e60-9944-78d068bee94a","Type":"ContainerStarted","Data":"2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674"}
May 25 14:15:09.711618 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.711478    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" event={"ID":"65804066-9483-4e60-9944-78d068bee94a","Type":"ContainerStarted","Data":"76f495fe32db74bd793c44abf89037dcac87124f1ec852955cfdac196cc669ee"}
May 25 14:15:09.712641 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.712620    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr_167b5203-12a1-4cde-a49a-596de4670058/storage-initializer/1.log"
May 25 14:15:09.712987 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.712961    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr_167b5203-12a1-4cde-a49a-596de4670058/storage-initializer/0.log"
May 25 14:15:09.713067 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.712991    2549 generic.go:358] "Generic (PLEG): container finished" podID="167b5203-12a1-4cde-a49a-596de4670058" containerID="e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7" exitCode=1
May 25 14:15:09.713067 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.713019    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr" event={"ID":"167b5203-12a1-4cde-a49a-596de4670058","Type":"ContainerDied","Data":"e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7"}
May 25 14:15:09.713067 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.713052    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr" event={"ID":"167b5203-12a1-4cde-a49a-596de4670058","Type":"ContainerDied","Data":"000154c758581f58bb86d8d0d4c492174f1988ed5fb2a25870afc02655b091ea"}
May 25 14:15:09.713067 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.713053    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"
May 25 14:15:09.713288 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.713070    2549 scope.go:117] "RemoveContainer" containerID="e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7"
May 25 14:15:09.720913 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.720896    2549 scope.go:117] "RemoveContainer" containerID="47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff"
May 25 14:15:09.728183 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.728166    2549 scope.go:117] "RemoveContainer" containerID="e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7"
May 25 14:15:09.728471 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:15:09.728447    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7\": container with ID starting with e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7 not found: ID does not exist" containerID="e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7"
May 25 14:15:09.728627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.728476    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7"} err="failed to get container status \"e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7\": rpc error: code = NotFound desc = could not find container \"e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7\": container with ID starting with e2f3843a875589a9168f32d6fd4c6452bd1be614b4f18265a0cf4bc9cd593bc7 not found: ID does not exist"
May 25 14:15:09.728627 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.728493    2549 scope.go:117] "RemoveContainer" containerID="47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff"
May 25 14:15:09.728774 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:15:09.728754    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff\": container with ID starting with 47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff not found: ID does not exist" containerID="47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff"
May 25 14:15:09.728834 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.728777    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff"} err="failed to get container status \"47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff\": rpc error: code = NotFound desc = could not find container \"47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff\": container with ID starting with 47e74d05aa13a35ec37c7285531d5ae23fac04826d11865a38ffdf2b8a52e8ff not found: ID does not exist"
May 25 14:15:09.787298 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.787186    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/167b5203-12a1-4cde-a49a-596de4670058-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"167b5203-12a1-4cde-a49a-596de4670058\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") "
May 25 14:15:09.787298 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.787243    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9hlw\" (UniqueName: \"kubernetes.io/projected/167b5203-12a1-4cde-a49a-596de4670058-kube-api-access-l9hlw\") pod \"167b5203-12a1-4cde-a49a-596de4670058\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") "
May 25 14:15:09.787298 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.787265    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls\") pod \"167b5203-12a1-4cde-a49a-596de4670058\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") "
May 25 14:15:09.787578 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.787321    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167b5203-12a1-4cde-a49a-596de4670058-kserve-provision-location\") pod \"167b5203-12a1-4cde-a49a-596de4670058\" (UID: \"167b5203-12a1-4cde-a49a-596de4670058\") "
May 25 14:15:09.787640 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.787602    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167b5203-12a1-4cde-a49a-596de4670058-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "167b5203-12a1-4cde-a49a-596de4670058" (UID: "167b5203-12a1-4cde-a49a-596de4670058"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:15:09.787640 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.787620    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167b5203-12a1-4cde-a49a-596de4670058-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "167b5203-12a1-4cde-a49a-596de4670058" (UID: "167b5203-12a1-4cde-a49a-596de4670058"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:15:09.787761 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.787716    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/167b5203-12a1-4cde-a49a-596de4670058-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:09.787761 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.787729    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/167b5203-12a1-4cde-a49a-596de4670058-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:09.789970 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.789942    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167b5203-12a1-4cde-a49a-596de4670058-kube-api-access-l9hlw" (OuterVolumeSpecName: "kube-api-access-l9hlw") pod "167b5203-12a1-4cde-a49a-596de4670058" (UID: "167b5203-12a1-4cde-a49a-596de4670058"). InnerVolumeSpecName "kube-api-access-l9hlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:15:09.789970 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.789956    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "167b5203-12a1-4cde-a49a-596de4670058" (UID: "167b5203-12a1-4cde-a49a-596de4670058"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:15:09.888459 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.888430    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hlw\" (UniqueName: \"kubernetes.io/projected/167b5203-12a1-4cde-a49a-596de4670058-kube-api-access-l9hlw\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:09.888459 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:09.888454    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/167b5203-12a1-4cde-a49a-596de4670058-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:15:10.046873 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:10.046780    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"]
May 25 14:15:10.051877 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:10.051847    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-49klr"]
May 25 14:15:10.496570 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:10.496537    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="167b5203-12a1-4cde-a49a-596de4670058" path="/var/lib/kubelet/pods/167b5203-12a1-4cde-a49a-596de4670058/volumes"
May 25 14:15:10.716958 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:10.716923    2549 generic.go:358] "Generic (PLEG): container finished" podID="65804066-9483-4e60-9944-78d068bee94a" containerID="2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674" exitCode=0
May 25 14:15:10.717109 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:10.717011    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" event={"ID":"65804066-9483-4e60-9944-78d068bee94a","Type":"ContainerDied","Data":"2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674"}
May 25 14:15:11.722181 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:11.722145    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" event={"ID":"65804066-9483-4e60-9944-78d068bee94a","Type":"ContainerStarted","Data":"e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad"}
May 25 14:15:11.722181 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:11.722180    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" event={"ID":"65804066-9483-4e60-9944-78d068bee94a","Type":"ContainerStarted","Data":"3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f"}
May 25 14:15:11.722697 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:11.722410    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:11.722697 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:11.722436    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:11.723594 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:11.723565    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 25 14:15:11.739900 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:11.739859    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podStartSLOduration=3.739848273 podStartE2EDuration="3.739848273s" podCreationTimestamp="2026-05-25 14:15:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:15:11.738884486 +0000 UTC m=+4261.777641608" watchObservedRunningTime="2026-05-25 14:15:11.739848273 +0000 UTC m=+4261.778605396"
May 25 14:15:12.728866 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:12.728821    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 25 14:15:17.730856 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:17.730828    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:15:17.731393 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:17.731366    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 25 14:15:27.732133 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:27.732092    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 25 14:15:37.732250 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:37.732206    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 25 14:15:47.731778 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:47.731736    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 25 14:15:57.731497 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:15:57.731412    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 25 14:16:07.731909 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:07.731859    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 25 14:16:17.731918 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:17.731878    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:16:18.166921 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:18.166879    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"]
May 25 14:16:18.167178 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:18.167145    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container" containerID="cri-o://3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f" gracePeriod=30
May 25 14:16:18.167369 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:18.167215    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kube-rbac-proxy" containerID="cri-o://e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad" gracePeriod=30
May 25 14:16:18.910007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:18.909972    2549 generic.go:358] "Generic (PLEG): container finished" podID="65804066-9483-4e60-9944-78d068bee94a" containerID="e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad" exitCode=2
May 25 14:16:18.910405 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:18.910050    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" event={"ID":"65804066-9483-4e60-9944-78d068bee94a","Type":"ContainerDied","Data":"e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad"}
May 25 14:16:19.218094 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.217993    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"]
May 25 14:16:19.218384 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.218367    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167b5203-12a1-4cde-a49a-596de4670058" containerName="storage-initializer"
May 25 14:16:19.218480 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.218387    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="167b5203-12a1-4cde-a49a-596de4670058" containerName="storage-initializer"
May 25 14:16:19.218480 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.218405    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167b5203-12a1-4cde-a49a-596de4670058" containerName="storage-initializer"
May 25 14:16:19.218480 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.218413    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="167b5203-12a1-4cde-a49a-596de4670058" containerName="storage-initializer"
May 25 14:16:19.218480 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.218478    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="167b5203-12a1-4cde-a49a-596de4670058" containerName="storage-initializer"
May 25 14:16:19.218690 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.218619    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="167b5203-12a1-4cde-a49a-596de4670058" containerName="storage-initializer"
May 25 14:16:19.221503 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.221481    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.224768 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.224744    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
May 25 14:16:19.224891 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.224771    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
May 25 14:16:19.233715 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.233690    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"]
May 25 14:16:19.312690 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.312650    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.312690 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.312694    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.312902 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.312746    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.312902 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.312774    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkk4\" (UniqueName: \"kubernetes.io/projected/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kube-api-access-gtkk4\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.413807 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.413762    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.413807 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.413813    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.414085 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.413861    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.414085 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.413896    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkk4\" (UniqueName: \"kubernetes.io/projected/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kube-api-access-gtkk4\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.414326 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.414307    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.414583 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.414561    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.416407 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.416374    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.424659 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.424634    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkk4\" (UniqueName: \"kubernetes.io/projected/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kube-api-access-gtkk4\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.530831 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.530727    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:19.665146 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.665110    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"]
May 25 14:16:19.668779 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:16:19.668751    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd82f7d_b01c_47c1_8371_d2689ceaa2e1.slice/crio-3f089d7011fb881311384dd556830c0a3ea6a3c333dffb59600cdfbf8736647d WatchSource:0}: Error finding container 3f089d7011fb881311384dd556830c0a3ea6a3c333dffb59600cdfbf8736647d: Status 404 returned error can't find the container with id 3f089d7011fb881311384dd556830c0a3ea6a3c333dffb59600cdfbf8736647d
May 25 14:16:19.914087 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.914046    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5" event={"ID":"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1","Type":"ContainerStarted","Data":"f2ca07f1a66d3a127aa56ea123e8b8eed9fecada0e3d42c053d7393b053428a7"}
May 25 14:16:19.914087 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:19.914088    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5" event={"ID":"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1","Type":"ContainerStarted","Data":"3f089d7011fb881311384dd556830c0a3ea6a3c333dffb59600cdfbf8736647d"}
May 25 14:16:22.614595 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.614571    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:16:22.638207 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.638153    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls\") pod \"65804066-9483-4e60-9944-78d068bee94a\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") "
May 25 14:16:22.638387 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.638325    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krg8q\" (UniqueName: \"kubernetes.io/projected/65804066-9483-4e60-9944-78d068bee94a-kube-api-access-krg8q\") pod \"65804066-9483-4e60-9944-78d068bee94a\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") "
May 25 14:16:22.638460 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.638407    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"65804066-9483-4e60-9944-78d068bee94a\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") "
May 25 14:16:22.638522 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.638464    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65804066-9483-4e60-9944-78d068bee94a-kserve-provision-location\") pod \"65804066-9483-4e60-9944-78d068bee94a\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") "
May 25 14:16:22.638586 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.638504    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-cabundle-cert\") pod \"65804066-9483-4e60-9944-78d068bee94a\" (UID: \"65804066-9483-4e60-9944-78d068bee94a\") "
May 25 14:16:22.639175 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.639143    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65804066-9483-4e60-9944-78d068bee94a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65804066-9483-4e60-9944-78d068bee94a" (UID: "65804066-9483-4e60-9944-78d068bee94a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:16:22.639175 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.639155    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65804066-9483-4e60-9944-78d068bee94a" (UID: "65804066-9483-4e60-9944-78d068bee94a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:16:22.639363 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.639254    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "65804066-9483-4e60-9944-78d068bee94a" (UID: "65804066-9483-4e60-9944-78d068bee94a"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:16:22.641250 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.641223    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65804066-9483-4e60-9944-78d068bee94a" (UID: "65804066-9483-4e60-9944-78d068bee94a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:16:22.641551 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.641528    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65804066-9483-4e60-9944-78d068bee94a-kube-api-access-krg8q" (OuterVolumeSpecName: "kube-api-access-krg8q") pod "65804066-9483-4e60-9944-78d068bee94a" (UID: "65804066-9483-4e60-9944-78d068bee94a"). InnerVolumeSpecName "kube-api-access-krg8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:16:22.739682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.739595    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:22.739682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.739625    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65804066-9483-4e60-9944-78d068bee94a-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:22.739682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.739636    2549 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65804066-9483-4e60-9944-78d068bee94a-cabundle-cert\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:22.739682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.739646    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65804066-9483-4e60-9944-78d068bee94a-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:22.739682 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.739655    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krg8q\" (UniqueName: \"kubernetes.io/projected/65804066-9483-4e60-9944-78d068bee94a-kube-api-access-krg8q\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:22.924134 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.924097    2549 generic.go:358] "Generic (PLEG): container finished" podID="65804066-9483-4e60-9944-78d068bee94a" containerID="3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f" exitCode=0
May 25 14:16:22.924329 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.924168    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" event={"ID":"65804066-9483-4e60-9944-78d068bee94a","Type":"ContainerDied","Data":"3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f"}
May 25 14:16:22.924329 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.924182    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"
May 25 14:16:22.924329 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.924205    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4" event={"ID":"65804066-9483-4e60-9944-78d068bee94a","Type":"ContainerDied","Data":"76f495fe32db74bd793c44abf89037dcac87124f1ec852955cfdac196cc669ee"}
May 25 14:16:22.924329 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.924227    2549 scope.go:117] "RemoveContainer" containerID="e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad"
May 25 14:16:22.925854 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.925831    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1/storage-initializer/0.log"
May 25 14:16:22.925971 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.925875    2549 generic.go:358] "Generic (PLEG): container finished" podID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" containerID="f2ca07f1a66d3a127aa56ea123e8b8eed9fecada0e3d42c053d7393b053428a7" exitCode=1
May 25 14:16:22.925971 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.925928    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5" event={"ID":"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1","Type":"ContainerDied","Data":"f2ca07f1a66d3a127aa56ea123e8b8eed9fecada0e3d42c053d7393b053428a7"}
May 25 14:16:22.943453 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.943431    2549 scope.go:117] "RemoveContainer" containerID="3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f"
May 25 14:16:22.968109 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.968088    2549 scope.go:117] "RemoveContainer" containerID="2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674"
May 25 14:16:22.978224 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.978197    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"]
May 25 14:16:22.988193 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.988173    2549 scope.go:117] "RemoveContainer" containerID="e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad"
May 25 14:16:22.988342 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.988257    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-6jgb4"]
May 25 14:16:22.988543 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:16:22.988524    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad\": container with ID starting with e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad not found: ID does not exist" containerID="e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad"
May 25 14:16:22.988609 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.988558    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad"} err="failed to get container status \"e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad\": rpc error: code = NotFound desc = could not find container \"e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad\": container with ID starting with e61a4ce7fff1420e6baaa49e1496343857377e7d56be49991f6fd547f0d97aad not found: ID does not exist"
May 25 14:16:22.988609 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.988584    2549 scope.go:117] "RemoveContainer" containerID="3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f"
May 25 14:16:22.988911 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:16:22.988889    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f\": container with ID starting with 3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f not found: ID does not exist" containerID="3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f"
May 25 14:16:22.989007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.988917    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f"} err="failed to get container status \"3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f\": rpc error: code = NotFound desc = could not find container \"3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f\": container with ID starting with 3ae4152b504e170ea1b00c2d7abfd627547610862cf0efccee95faf8616f034f not found: ID does not exist"
May 25 14:16:22.989007 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.988934    2549 scope.go:117] "RemoveContainer" containerID="2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674"
May 25 14:16:22.989423 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:16:22.989408    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674\": container with ID starting with 2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674 not found: ID does not exist" containerID="2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674"
May 25 14:16:22.989481 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:22.989427    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674"} err="failed to get container status \"2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674\": rpc error: code = NotFound desc = could not find container \"2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674\": container with ID starting with 2064488ad40488b024cbf4df36374370ce5fa88ecfe441ed059359451052f674 not found: ID does not exist"
May 25 14:16:23.931015 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:23.930989    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1/storage-initializer/0.log"
May 25 14:16:23.931413 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:23.931060    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5" event={"ID":"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1","Type":"ContainerStarted","Data":"9c9fda147d1250a7dbef3104a39816426204e54f7fb6e1a403a10a1298beec1b"}
May 25 14:16:24.496675 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:24.496631    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65804066-9483-4e60-9944-78d068bee94a" path="/var/lib/kubelet/pods/65804066-9483-4e60-9944-78d068bee94a/volumes"
May 25 14:16:27.942310 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:27.942256    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1/storage-initializer/1.log"
May 25 14:16:27.942656 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:27.942642    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1/storage-initializer/0.log"
May 25 14:16:27.942716 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:27.942675    2549 generic.go:358] "Generic (PLEG): container finished" podID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" containerID="9c9fda147d1250a7dbef3104a39816426204e54f7fb6e1a403a10a1298beec1b" exitCode=1
May 25 14:16:27.942716 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:27.942704    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5" event={"ID":"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1","Type":"ContainerDied","Data":"9c9fda147d1250a7dbef3104a39816426204e54f7fb6e1a403a10a1298beec1b"}
May 25 14:16:27.942821 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:27.942730    2549 scope.go:117] "RemoveContainer" containerID="f2ca07f1a66d3a127aa56ea123e8b8eed9fecada0e3d42c053d7393b053428a7"
May 25 14:16:27.943189 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:27.943164    2549 scope.go:117] "RemoveContainer" containerID="f2ca07f1a66d3a127aa56ea123e8b8eed9fecada0e3d42c053d7393b053428a7"
May 25 14:16:27.954119 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:16:27.954075    2549 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_kserve-ci-e2e-test_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1_0 in pod sandbox 3f089d7011fb881311384dd556830c0a3ea6a3c333dffb59600cdfbf8736647d from index: no such id: 'f2ca07f1a66d3a127aa56ea123e8b8eed9fecada0e3d42c053d7393b053428a7'" containerID="f2ca07f1a66d3a127aa56ea123e8b8eed9fecada0e3d42c053d7393b053428a7"
May 25 14:16:27.954220 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:16:27.954131    2549 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_kserve-ci-e2e-test_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1_0 in pod sandbox 3f089d7011fb881311384dd556830c0a3ea6a3c333dffb59600cdfbf8736647d from index: no such id: 'f2ca07f1a66d3a127aa56ea123e8b8eed9fecada0e3d42c053d7393b053428a7'; Skipping pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_kserve-ci-e2e-test(3cd82f7d-b01c-47c1-8371-d2689ceaa2e1)\"" logger="UnhandledError"
May 25 14:16:27.955450 ip-10-0-129-55 kubenswrapper[2549]: E0525 14:16:27.955430    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_kserve-ci-e2e-test(3cd82f7d-b01c-47c1-8371-d2689ceaa2e1)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5" podUID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1"
May 25 14:16:28.946385 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:28.946358    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1/storage-initializer/1.log"
May 25 14:16:29.249340 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.249222    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"]
May 25 14:16:29.376776 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.376752    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1/storage-initializer/1.log"
May 25 14:16:29.376889 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.376815    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:29.489631 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.489596    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") "
May 25 14:16:29.489805 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.489653    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-proxy-tls\") pod \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") "
May 25 14:16:29.489805 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.489708    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtkk4\" (UniqueName: \"kubernetes.io/projected/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kube-api-access-gtkk4\") pod \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") "
May 25 14:16:29.489890 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.489871    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kserve-provision-location\") pod \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\" (UID: \"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1\") "
May 25 14:16:29.489945 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.489916    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" (UID: "3cd82f7d-b01c-47c1-8371-d2689ceaa2e1"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 14:16:29.490121 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.490098    2549 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:29.490239 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.490132    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" (UID: "3cd82f7d-b01c-47c1-8371-d2689ceaa2e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 14:16:29.491923 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.491900    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kube-api-access-gtkk4" (OuterVolumeSpecName: "kube-api-access-gtkk4") pod "3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" (UID: "3cd82f7d-b01c-47c1-8371-d2689ceaa2e1"). InnerVolumeSpecName "kube-api-access-gtkk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 14:16:29.491999 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.491902    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" (UID: "3cd82f7d-b01c-47c1-8371-d2689ceaa2e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 14:16:29.591166 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.591087    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtkk4\" (UniqueName: \"kubernetes.io/projected/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kube-api-access-gtkk4\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:29.591166 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.591117    2549 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-kserve-provision-location\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:29.591166 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.591127    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1-proxy-tls\") on node \"ip-10-0-129-55.ec2.internal\" DevicePath \"\""
May 25 14:16:29.950560 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.950534    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5_3cd82f7d-b01c-47c1-8371-d2689ceaa2e1/storage-initializer/1.log"
May 25 14:16:29.951034 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.950590    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5" event={"ID":"3cd82f7d-b01c-47c1-8371-d2689ceaa2e1","Type":"ContainerDied","Data":"3f089d7011fb881311384dd556830c0a3ea6a3c333dffb59600cdfbf8736647d"}
May 25 14:16:29.951034 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.950622    2549 scope.go:117] "RemoveContainer" containerID="9c9fda147d1250a7dbef3104a39816426204e54f7fb6e1a403a10a1298beec1b"
May 25 14:16:29.951034 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.950670    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"
May 25 14:16:29.997524 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:29.997494    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"]
May 25 14:16:30.006114 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:30.006092    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-x8mw5"]
May 25 14:16:30.496395 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:16:30.496358    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" path="/var/lib/kubelet/pods/3cd82f7d-b01c-47c1-8371-d2689ceaa2e1/volumes"
May 25 14:17:05.908237 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:05.908208    2549 ???:1] "http: TLS handshake error from 10.0.129.55:50160: EOF"
May 25 14:17:05.915667 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:05.915643    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrxhv_6ec74b53-fbd6-407f-a5c8-cefb6f613f10/global-pull-secret-syncer/0.log"
May 25 14:17:06.069638 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:06.069608    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8rwnv_feaa61e8-ce75-4474-8a0c-bc149dcb8074/konnectivity-agent/0.log"
May 25 14:17:06.132377 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:06.132349    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-55.ec2.internal_95dfbc508c808d021aadda17116338a3/haproxy/0.log"
May 25 14:17:09.927686 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:09.927653    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqv4n_f4517545-b70c-47d7-abfb-e1af57d3d3fb/node-exporter/0.log"
May 25 14:17:09.959360 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:09.959328    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqv4n_f4517545-b70c-47d7-abfb-e1af57d3d3fb/kube-rbac-proxy/0.log"
May 25 14:17:09.987106 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:09.987080    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqv4n_f4517545-b70c-47d7-abfb-e1af57d3d3fb/init-textfile/0.log"
May 25 14:17:12.898550 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898513    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"]
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898749    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="storage-initializer"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898759    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="storage-initializer"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898767    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898773    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898789    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" containerName="storage-initializer"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898794    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" containerName="storage-initializer"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898800    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" containerName="storage-initializer"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898805    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" containerName="storage-initializer"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898812    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kube-rbac-proxy"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898817    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kube-rbac-proxy"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898855    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kube-rbac-proxy"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898862    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" containerName="storage-initializer"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898869    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="3cd82f7d-b01c-47c1-8371-d2689ceaa2e1" containerName="storage-initializer"
May 25 14:17:12.899006 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.898875    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="65804066-9483-4e60-9944-78d068bee94a" containerName="kserve-container"
May 25 14:17:12.900593 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.900573    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:12.903265 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.903245    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-45q49\"/\"openshift-service-ca.crt\""
May 25 14:17:12.903379 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.903246    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-45q49\"/\"kube-root-ca.crt\""
May 25 14:17:12.904321 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.904301    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-45q49\"/\"default-dockercfg-k66pn\""
May 25 14:17:12.917395 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:12.917370    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"]
May 25 14:17:13.003727 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.003683    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-lib-modules\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.003920 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.003736    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28rb\" (UniqueName: \"kubernetes.io/projected/bdc425d8-30c7-4d46-95e7-46a668158272-kube-api-access-b28rb\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.003920 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.003814    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-proc\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.003920 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.003882    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-podres\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.003920 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.003910    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-sys\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104665 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104627    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-lib-modules\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104665 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104671    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b28rb\" (UniqueName: \"kubernetes.io/projected/bdc425d8-30c7-4d46-95e7-46a668158272-kube-api-access-b28rb\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104932 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104699    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-proc\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104932 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104720    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-podres\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104932 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104741    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-sys\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104932 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104807    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-sys\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104932 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104806    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-lib-modules\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104932 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104807    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-proc\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.104932 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.104836    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bdc425d8-30c7-4d46-95e7-46a668158272-podres\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.113637 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.113612    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28rb\" (UniqueName: \"kubernetes.io/projected/bdc425d8-30c7-4d46-95e7-46a668158272-kube-api-access-b28rb\") pod \"perf-node-gather-daemonset-dpwmb\" (UID: \"bdc425d8-30c7-4d46-95e7-46a668158272\") " pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.210255 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.210164    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:13.336841 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.336661    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"]
May 25 14:17:13.339812 ip-10-0-129-55 kubenswrapper[2549]: W0525 14:17:13.339784    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbdc425d8_30c7_4d46_95e7_46a668158272.slice/crio-c34adbf384568f880823fc11286180797bfb53f53d105a9c66b10bae745eb638 WatchSource:0}: Error finding container c34adbf384568f880823fc11286180797bfb53f53d105a9c66b10bae745eb638: Status 404 returned error can't find the container with id c34adbf384568f880823fc11286180797bfb53f53d105a9c66b10bae745eb638
May 25 14:17:13.925823 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.925794    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cflbv_0e7829ab-3942-479d-9ddf-12c4352e88d7/dns/0.log"
May 25 14:17:13.946572 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:13.946544    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cflbv_0e7829ab-3942-479d-9ddf-12c4352e88d7/kube-rbac-proxy/0.log"
May 25 14:17:14.044544 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:14.044515    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6hzsz_1363056f-5cf2-4736-8032-7cbc4cd602e1/dns-node-resolver/0.log"
May 25 14:17:14.067361 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:14.067329    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb" event={"ID":"bdc425d8-30c7-4d46-95e7-46a668158272","Type":"ContainerStarted","Data":"788b48f2a8dd26227f413090eac7ef1ae0fbb36b0a577d27a1456b0417afa5a2"}
May 25 14:17:14.067361 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:14.067362    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb" event={"ID":"bdc425d8-30c7-4d46-95e7-46a668158272","Type":"ContainerStarted","Data":"c34adbf384568f880823fc11286180797bfb53f53d105a9c66b10bae745eb638"}
May 25 14:17:14.067571 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:14.067475    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:14.091247 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:14.091200    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb" podStartSLOduration=2.091185516 podStartE2EDuration="2.091185516s" podCreationTimestamp="2026-05-25 14:17:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 14:17:14.089807312 +0000 UTC m=+4384.128564447" watchObservedRunningTime="2026-05-25 14:17:14.091185516 +0000 UTC m=+4384.129942640"
May 25 14:17:14.572442 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:14.572412    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pqbcw_2b9ff95a-a2af-4375-beea-7e288a65ab11/node-ca/0.log"
May 25 14:17:15.664845 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:15.664811    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fwdbr_386b85a9-b8f6-4f1d-b0c8-e45da5a1b514/serve-healthcheck-canary/0.log"
May 25 14:17:16.133005 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:16.132980    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvqmv_25c33450-733b-42b2-bdaa-cf1c9734355d/kube-rbac-proxy/0.log"
May 25 14:17:16.155770 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:16.155744    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvqmv_25c33450-733b-42b2-bdaa-cf1c9734355d/exporter/0.log"
May 25 14:17:16.177917 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:16.177894    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvqmv_25c33450-733b-42b2-bdaa-cf1c9734355d/extractor/0.log"
May 25 14:17:18.373670 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:18.373637    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-6b6fw_da4a53a7-6605-4b9d-8d3e-828d646ad77c/server/0.log"
May 25 14:17:18.698242 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:18.698137    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-4w4z2_659ee578-5137-48d8-8315-f9d40c932695/s3-tls-init-custom/0.log"
May 25 14:17:18.801088 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:18.801061    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-7fd5766db9-2dt72_07690b93-a4a3-4b9e-94ff-5ae135d57aef/seaweedfs-tls-serving/0.log"
May 25 14:17:20.080965 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:20.080934    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-45q49/perf-node-gather-daemonset-dpwmb"
May 25 14:17:24.175110 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.175078    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2mzsn_8c6d0a9e-64c4-44de-8429-196cdbb51c51/kube-multus/0.log"
May 25 14:17:24.417416 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.417389    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw2vp_218a9570-215b-4ca9-8d61-f9f39f8295e5/kube-multus-additional-cni-plugins/0.log"
May 25 14:17:24.469406 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.469368    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw2vp_218a9570-215b-4ca9-8d61-f9f39f8295e5/egress-router-binary-copy/0.log"
May 25 14:17:24.486725 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.486697    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw2vp_218a9570-215b-4ca9-8d61-f9f39f8295e5/cni-plugins/0.log"
May 25 14:17:24.510117 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.510087    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw2vp_218a9570-215b-4ca9-8d61-f9f39f8295e5/bond-cni-plugin/0.log"
May 25 14:17:24.534016 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.533989    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw2vp_218a9570-215b-4ca9-8d61-f9f39f8295e5/routeoverride-cni/0.log"
May 25 14:17:24.558458 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.558431    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw2vp_218a9570-215b-4ca9-8d61-f9f39f8295e5/whereabouts-cni-bincopy/0.log"
May 25 14:17:24.608506 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.608477    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lw2vp_218a9570-215b-4ca9-8d61-f9f39f8295e5/whereabouts-cni/0.log"
May 25 14:17:24.836632 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.836602    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kmndm_cf5b6105-22f6-4f92-9cd2-5078d464a323/network-metrics-daemon/0.log"
May 25 14:17:24.857010 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:24.856984    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kmndm_cf5b6105-22f6-4f92-9cd2-5078d464a323/kube-rbac-proxy/0.log"
May 25 14:17:26.292743 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.292712    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-controller/0.log"
May 25 14:17:26.313450 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.313421    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/0.log"
May 25 14:17:26.330830 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.330806    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovn-acl-logging/1.log"
May 25 14:17:26.350065 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.350037    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/kube-rbac-proxy-node/0.log"
May 25 14:17:26.371721 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.371689    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/kube-rbac-proxy-ovn-metrics/0.log"
May 25 14:17:26.415055 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.415026    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/northd/0.log"
May 25 14:17:26.495816 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.495771    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/nbdb/0.log"
May 25 14:17:26.520223 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.520195    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/sbdb/0.log"
May 25 14:17:26.622514 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:26.622483    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qx45s_22897fc3-4085-4592-92b7-ba00a4a8dfaa/ovnkube-controller/0.log"
May 25 14:17:27.723369 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:27.723336    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qnwqv_10569d7e-c62f-4724-b920-daf76ac7365f/network-check-target-container/0.log"
May 25 14:17:28.650636 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:28.650606    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2hc5v_93dafc65-1e53-4bf2-a21c-c8da486adaea/iptables-alerter/0.log"
May 25 14:17:29.363510 ip-10-0-129-55 kubenswrapper[2549]: I0525 14:17:29.363479    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xs46z_5a934d1e-67ac-490f-b795-9203848f808b/tuned/0.log"